2024-09-16T03:27:57.787 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T03:27:57.806 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T03:27:57.891 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894 branch: squid description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905894' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCNbHsjzHuQ2cUQigTd2Zn5fjtng3xHz/bfDQyfr/jeGKA1bXp39Tfl5sQ2PpbgVsy/ASTUOhdae+rotcKAQV38= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFz+8VMv6RaQlzLP67KTQeG6p329gwORwDSlf+NDA4pPTK9wjtc80tTfi7oN+nmEpfW8EOxoRef50PsqFIqaTro= smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJdfUzuR24onZlhr0YvfJhNnZDtOkcOLo7drXQ0vCtlRfm/D3py8sJcHIwSuWQBhQj87lC4719RLKwOSc03Sph0= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDAv/poHrnNt/ymxP7JkuPVKBmUmzvhSBZyn04dchPsF9uDmufTWPJ/TfJrKboV+GN293qv+Gg8+vxDIiU9Ruaw= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T03:27:57.891 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T03:27:57.892 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T03:27:57.892 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T03:27:57.894 INFO:teuthology.task.internal:Checking packages... 2024-09-16T03:27:57.915 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T03:27:57.915 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T03:27:57.915 INFO:teuthology.packaging:ref: None 2024-09-16T03:27:57.915 INFO:teuthology.packaging:tag: None 2024-09-16T03:27:57.915 INFO:teuthology.packaging:branch: squid 2024-09-16T03:27:57.915 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:27:57.916 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-16T03:27:58.150 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-16T03:27:58.151 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T03:27:58.158 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T03:27:58.158 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T03:27:58.180 INFO:teuthology.task.internal:Saving configuration 2024-09-16T03:27:58.196 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T03:27:58.202 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T03:27:58.228 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:22:32.739856', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCNbHsjzHuQ2cUQigTd2Zn5fjtng3xHz/bfDQyfr/jeGKA1bXp39Tfl5sQ2PpbgVsy/ASTUOhdae+rotcKAQV38='} 2024-09-16T03:27:58.247 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:22:32.736066', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFz+8VMv6RaQlzLP67KTQeG6p329gwORwDSlf+NDA4pPTK9wjtc80tTfi7oN+nmEpfW8EOxoRef50PsqFIqaTro='} 2024-09-16T03:27:58.271 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:22:32.737527', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJdfUzuR24onZlhr0YvfJhNnZDtOkcOLo7drXQ0vCtlRfm/D3py8sJcHIwSuWQBhQj87lC4719RLKwOSc03Sph0='} 2024-09-16T03:27:58.294 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:22:32.738701', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDAv/poHrnNt/ymxP7JkuPVKBmUmzvhSBZyn04dchPsF9uDmufTWPJ/TfJrKboV+GN293qv+Gg8+vxDIiU9Ruaw='} 2024-09-16T03:27:58.295 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T03:27:58.301 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-16T03:27:58.301 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-16T03:27:58.301 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-16T03:27:58.301 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-16T03:27:58.301 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T03:27:58.414 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb6df59bac0>, signals=[15]) 2024-09-16T03:27:58.414 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T03:27:58.420 INFO:teuthology.task.internal:Opening connections... 2024-09-16T03:27:58.420 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:27:58.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:58.499 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:27:58.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:58.575 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:27:58.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:58.648 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:27:58.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:58.722 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T03:27:58.728 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-09-16T03:27:58.746 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-09-16T03:27:58.747 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-09-16T03:27:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2024-09-16T03:27:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9" 2024-09-16T03:27:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2024-09-16T03:27:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:27:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9" 2024-09-16T03:27:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:27:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:27:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-09-16T03:27:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:27:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:27:58.805 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:27:58.805 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:27:58.805 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:27:58.805 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:27:58.805 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-09-16T03:27:58.839 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-09-16T03:27:58.857 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-09-16T03:27:58.857 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-09-16T03:27:58.915 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:27:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:27:58.917 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:27:58.917 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-09-16T03:27:58.955 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-09-16T03:27:58.974 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-09-16T03:27:58.974 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:27:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:27:59.034 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:27:59.034 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:27:59.034 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:27:59.034 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-09-16T03:27:59.064 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-09-16T03:27:59.084 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-09-16T03:27:59.084 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-09-16T03:27:59.142 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:27:59.143 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:27:59.144 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-09-16T03:27:59.173 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T03:27:59.182 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T03:27:59.192 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T03:27:59.192 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:27:59.195 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:27:59.197 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:27:59.200 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:27:59.218 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T03:27:59.224 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T03:27:59.224 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:27:59.252 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:27:59.255 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:27:59.257 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:27:59.417 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T03:27:59.524 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T03:27:59.539 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T03:27:59.539 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T03:27:59.539 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:27:59.540 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:27:59.540 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:27:59.541 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:27:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:27:59.558 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-09-16T03:27:59.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:27:59.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:27:59.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:27:59.561 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-09-16T03:27:59.561 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-09-16T03:27:59.562 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-09-16T03:27:59.616 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:27:59.616 INFO:teuthology.task.kernel:Running kernel on smithi076: 5.14.0-503.el9.x86_64 2024-09-16T03:27:59.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum install -y kernel 2024-09-16T03:27:59.618 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:27:59.618 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-503.el9.x86_64 2024-09-16T03:27:59.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-09-16T03:27:59.620 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:27:59.620 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:27:59.620 INFO:teuthology.task.kernel:Running kernel on smithi119: 5.14.0-503.el9.x86_64 2024-09-16T03:27:59.620 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2024-09-16T03:27:59.621 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-503.el9.x86_64 2024-09-16T03:27:59.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-09-16T03:28:01.967 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-16T03:28:01.980 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-16T03:28:03.229 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-09-16T03:28:03.909 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-16T03:28:04.152 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.2 MB 00:02 2024-09-16T03:28:05.122 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-16T03:28:05.464 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 20 MB 00:02 2024-09-16T03:28:07.120 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-16T03:28:09.076 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.5 MB 00:01 2024-09-16T03:28:09.289 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-16T03:28:09.907 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-16T03:28:10.702 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 19 kB 00:00 2024-09-16T03:28:10.915 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 19 kB 00:00 2024-09-16T03:28:11.487 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-16T03:28:11.948 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-16T03:28:12.168 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-16T03:28:12.199 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-16T03:28:12.854 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 115 kB/s | 19 kB 00:00 2024-09-16T03:28:13.586 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-16T03:28:14.026 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-16T03:28:17.445 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T03:28:17.649 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-16T03:28:18.942 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-16T03:28:19.490 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T03:28:19.493 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:28:19.493 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:28:19.585 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:28:19.593 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:28:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-09-16T03:28:19.593 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout:Install 4 Packages 2024-09-16T03:28:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:28:19.595 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 79 M 2024-09-16T03:28:19.595 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 126 M 2024-09-16T03:28:19.595 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:28:19.647 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:28:19.647 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:28:19.731 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:28:19.741 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:28:19.741 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-16T03:28:19.741 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:28:19.742 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:28:19.742 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:28:19.742 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-16T03:28:19.742 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:28:19.742 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:28:19.742 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:28:19.742 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:28:19.742 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:28:19.742 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:28:19.742 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-09-16T03:28:19.743 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:28:19.743 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 79 M 2024-09-16T03:28:19.743 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 126 M 2024-09-16T03:28:19.743 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:28:19.917 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 333 kB/s | 23 kB 00:00 2024-09-16T03:28:19.996 INFO:teuthology.orchestra.run.smithi076.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 184 kB/s | 23 kB 00:00 2024-09-16T03:28:20.965 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:28:20.965 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:28:21.039 INFO:teuthology.orchestra.run.smithi076.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 27 MB/s | 29 MB 00:01 2024-09-16T03:28:21.050 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:28:21.061 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:28:21.061 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-16T03:28:21.061 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:28:21.061 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:28:21.061 INFO:teuthology.orchestra.run.smithi119.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:28:21.061 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-16T03:28:21.061 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:28:21.061 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:28:21.062 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:28:21.062 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:28:21.062 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:28:21.062 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:28:21.062 INFO:teuthology.orchestra.run.smithi119.stdout:Install 4 Packages 2024-09-16T03:28:21.062 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:28:21.062 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 79 M 2024-09-16T03:28:21.063 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 126 M 2024-09-16T03:28:21.063 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:28:21.212 INFO:teuthology.orchestra.run.smithi119.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 223 kB/s | 23 kB 00:00 2024-09-16T03:28:21.375 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 23 MB/s | 35 MB 00:01 2024-09-16T03:28:21.521 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:28:21.521 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:28:21.596 INFO:teuthology.orchestra.run.smithi119.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-16T03:28:21.610 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:28:21.622 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:28:21.622 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-09-16T03:28:21.623 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:28:21.623 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:28:21.623 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:28:21.623 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T03:28:21.623 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:28:21.623 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:28:21.624 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:28:21.624 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:28:21.624 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:28:21.624 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:28:21.624 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-09-16T03:28:21.624 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:28:21.625 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 79 M 2024-09-16T03:28:21.626 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 126 M 2024-09-16T03:28:21.626 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:28:21.714 INFO:teuthology.orchestra.run.smithi076.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 8.5 MB/s | 16 MB 00:01 2024-09-16T03:28:21.766 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 241 kB/s | 23 kB 00:00 2024-09-16T03:28:21.859 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 15 MB/s | 29 MB 00:01 2024-09-16T03:28:21.980 INFO:teuthology.orchestra.run.smithi119.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 37 MB/s | 29 MB 00:00 2024-09-16T03:28:22.280 INFO:teuthology.orchestra.run.smithi119.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 30 MB/s | 35 MB 00:01 2024-09-16T03:28:22.280 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:28:22.280 INFO:teuthology.orchestra.run.smithi119.stdout:Total 65 MB/s | 79 MB 00:01 2024-09-16T03:28:22.342 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): kernel-core-5.14.0-505.el9.x86_64.rpm 6.3 MB/s | 16 MB 00:02 2024-09-16T03:28:22.342 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:28:22.343 INFO:teuthology.orchestra.run.smithi121.stdout:Total 30 MB/s | 79 MB 00:02 2024-09-16T03:28:22.905 INFO:teuthology.orchestra.run.smithi076.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 11 MB/s | 35 MB 00:03 2024-09-16T03:28:22.906 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:28:22.906 INFO:teuthology.orchestra.run.smithi076.stdout:Total 24 MB/s | 79 MB 00:03 2024-09-16T03:28:22.961 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:28:23.025 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:28:23.312 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:28:23.312 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:28:23.371 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:28:23.371 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:28:23.578 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:28:23.952 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:28:23.952 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:28:24.258 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:28:24.258 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:28:24.322 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:28:24.322 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:28:24.668 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.2 MB/s | 16 MB 00:02 2024-09-16T03:28:24.912 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:28:24.912 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:28:26.218 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.4 MB/s | 29 MB 00:04 2024-09-16T03:28:26.693 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:28:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:28:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:28:27.371 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:28:27.456 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:28:27.502 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:28:27.584 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:28:27.844 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 5.6 MB/s | 35 MB 00:06 2024-09-16T03:28:27.844 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:28:27.844 INFO:teuthology.orchestra.run.smithi103.stdout:Total 13 MB/s | 79 MB 00:06 2024-09-16T03:28:28.019 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:28:28.096 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:28:28.522 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:28:28.905 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:28:28.905 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:28:29.865 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:28:29.866 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:28:30.245 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:28:30.375 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:28:30.392 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:28:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:28:30.898 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:28:31.046 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:28:32.357 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:28:33.080 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:28:33.169 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:28:35.249 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:28:35.325 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:28:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:28:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:28:35.889 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:28:35.929 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:28:35.991 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:28:36.121 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:28:40.172 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:28:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:28:40.754 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:28:41.006 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:28:41.155 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:28:46.018 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:18.526 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:18.566 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:29:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:29:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:29:18.963 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:18.963 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:29:18.963 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:29:18.963 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:29:19.053 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:19.099 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:19.503 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:19.504 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:29:19.504 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:29:19.504 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:29:19.776 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:19.776 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:29:19.777 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:29:19.777 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:19.777 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:29:19.777 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:29:19.777 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:29:19.777 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:29:19.777 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:29:19.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:19.809 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:29:19.809 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:29:19.810 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:19.810 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:29:19.810 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:29:19.810 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:29:19.810 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:29:19.810 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:29:20.135 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:29:20.208 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:29:20.218 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:20.219 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.219 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.219 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:29:20.219 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:29:20.219 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-16T03:29:20.219 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:29:20.219 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:29:20.278 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:20.278 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-09-16T03:29:20.300 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:20.300 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.300 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.300 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:29:20.300 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:29:20.300 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-16T03:29:20.300 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:29:20.300 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 82.el9 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8274670 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:29:20.316 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:29:20.317 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:29:20.318 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:29:20.318 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:29:20.359 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:20.359 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2024-09-16T03:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:29:20.402 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2024-09-16T03:29:20.402 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2024-09-16T03:29:20.402 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.06 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 82.el9 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Group : Unspecified 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 8274670 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : builder@centos.org 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : CentOS 2024-09-16T03:29:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:29:20.404 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2024-09-16T03:29:20.404 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2024-09-16T03:29:20.404 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:29:20.404 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:29:20.404 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:29:20.404 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:29:20.404 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2024-09-16T03:29:20.404 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:29:20.404 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:29:20.405 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:29:20.405 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:29:20.744 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:29:20.828 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:20.828 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.828 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.828 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:29:20.828 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:29:20.828 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T03:29:20.829 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:29:20.829 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:29:20.887 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:20.887 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2-tools 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2-tools 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.06 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 82.el9 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:Group : Unspecified 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 8274670 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:29:20.920 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:29:20.921 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:29:20.921 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:29:20.921 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : builder@centos.org 2024-09-16T03:29:20.921 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2024-09-16T03:29:20.921 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:29:20.921 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Support tools for GRUB. 2024-09-16T03:29:20.921 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2024-09-16T03:29:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:29:20.921 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:29:20.921 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:29:20.922 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:29:20.922 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2024-09-16T03:29:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:29:20.922 INFO:teuthology.orchestra.run.smithi076.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:29:20.923 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:29:20.923 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:29:20.923 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:29:20.985 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-09-16T03:29:21.026 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-09-16T03:29:21.632 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2024-09-16T03:29:22.252 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:29:22.277 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-09-16T03:29:22.280 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-16T03:29:22.295 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.NyIVriv25N 2024-09-16T03:29:22.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NyIVriv25N 2024-09-16T03:29:22.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.NyIVriv25N 2024-09-16T03:29:22.375 INFO:teuthology.orchestra.run.smithi119.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:29:22.404 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-09-16T03:29:22.406 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-09-16T03:29:22.422 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.OFhmPu0WWY 2024-09-16T03:29:22.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OFhmPu0WWY 2024-09-16T03:29:22.489 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.OFhmPu0WWY 2024-09-16T03:29:22.510 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.NyIVriv25N is 6KB 2024-09-16T03:29:22.559 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.NyIVriv25N 2024-09-16T03:29:22.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:29:22.627 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.OFhmPu0WWY is 6KB 2024-09-16T03:29:22.637 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:29:22.638 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:29:22.638 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:29:22.638 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:29:22.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:29:22.676 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.OFhmPu0WWY 2024-09-16T03:29:22.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:29:22.763 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:29:22.763 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:29:22.763 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:29:22.763 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:29:22.765 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:29:22.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-09-16T03:29:22.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-09-16T03:29:22.981 INFO:teuthology.orchestra.run.smithi076.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:29:23.008 INFO:teuthology.orchestra.run.smithi076.stderr:done 2024-09-16T03:29:23.011 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-09-16T03:29:23.026 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.2XxTvJUiIN 2024-09-16T03:29:23.027 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2XxTvJUiIN 2024-09-16T03:29:23.093 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.2XxTvJUiIN 2024-09-16T03:29:23.232 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.2XxTvJUiIN is 6KB 2024-09-16T03:29:23.280 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.2XxTvJUiIN 2024-09-16T03:29:23.295 DEBUG:teuthology.orchestra.run.smithi076:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:29:23.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:29:23.521 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2024-09-16T03:29:24.279 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:24.319 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:24.599 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:24.599 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:29:24.599 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:29:24.599 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:29:25.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:29:25.435 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:29:25.435 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:29:25.435 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:25.436 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:29:25.436 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:29:25.436 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:29:25.436 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:29:25.436 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:29:25.776 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:29:25.865 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:25.865 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:29:25.865 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:29:25.865 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:29:25.865 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:29:25.866 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T03:29:25.866 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:29:25.866 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:29:25.924 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:29:25.924 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:29:25.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-09-16T03:29:25.963 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-09-16T03:29:25.963 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-09-16T03:29:25.963 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-09-16T03:29:25.963 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 82.el9 2024-09-16T03:29:25.963 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-09-16T03:29:25.963 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8274670 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-09-16T03:29:25.964 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:29:25.965 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:29:25.965 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:29:25.965 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:29:25.965 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-09-16T03:29:25.965 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:29:25.965 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:29:25.966 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:29:25.966 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:29:25.966 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:29:26.588 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-09-16T03:29:27.919 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:29:27.946 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-09-16T03:29:27.953 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-09-16T03:29:27.968 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.UeQOztyVmk 2024-09-16T03:29:27.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UeQOztyVmk 2024-09-16T03:29:28.035 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.UeQOztyVmk 2024-09-16T03:29:28.167 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.UeQOztyVmk is 6KB 2024-09-16T03:29:28.188 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.UeQOztyVmk 2024-09-16T03:29:28.203 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:29:28.265 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:29:28.265 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:29:28.265 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:29:28.265 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:29:28.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:29:28.431 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-09-16T03:29:52.817 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-16T03:29:52.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-16T03:29:52.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:29:52.921 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-16T03:29:52.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-16T03:29:52.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:29:53.524 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T03:29:53.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T03:29:53.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:29:58.437 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T03:29:58.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T03:29:58.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:11.137 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-09-16T03:30:11.389 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-09-16T03:30:11.901 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-16T03:30:16.765 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-09-16T03:30:20.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-16T03:30:20.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:20.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-16T03:30:20.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:20.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T03:30:20.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:23.197 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-09-16T03:30:23.457 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-09-16T03:30:23.969 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-16T03:30:25.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T03:30:25.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:28.829 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-09-16T03:30:35.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-16T03:30:35.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:35.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-16T03:30:35.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:35.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T03:30:35.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:39.037 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-16T03:30:41.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T03:30:41.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:54.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T03:30:54.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:31:07.185 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-09-16T03:31:07.286 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-09-16T03:31:07.640 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-16T03:31:07.640 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:31:07.640 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-09-16T03:31:07.655 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:31:07.655 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:31:07.655 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:31:07.655 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:31:07.867 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-16T03:31:07.868 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:31:07.868 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-09-16T03:31:07.887 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:31:07.887 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:31:07.887 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:31:07.887 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:31:08.656 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-16T03:31:08.656 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:31:08.751 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2024-09-16T03:31:08.771 DEBUG:teuthology.parallel:result is None 2024-09-16T03:31:08.888 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-16T03:31:08.889 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:31:08.959 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-09-16T03:31:08.981 DEBUG:teuthology.parallel:result is None 2024-09-16T03:31:25.923 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-09-16T03:31:26.295 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T03:31:26.295 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:31:26.296 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-09-16T03:31:26.310 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:31:26.311 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:31:26.311 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:31:26.311 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:31:27.312 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T03:31:27.312 DEBUG:teuthology.orchestra.run.smithi076:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:31:27.389 INFO:teuthology.orchestra.run.smithi076.stdout:ttyS1 2024-09-16T03:31:27.412 DEBUG:teuthology.parallel:result is None 2024-09-16T03:31:41.078 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:31:56.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T03:31:56.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:31:56.728 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-09-16T03:31:57.078 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T03:31:57.078 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:31:57.079 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-09-16T03:31:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:31:57.095 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:31:57.095 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:31:57.095 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:31:58.097 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T03:31:58.097 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:31:58.176 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-09-16T03:31:58.193 DEBUG:teuthology.parallel:result is None 2024-09-16T03:31:58.194 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T03:31:58.201 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T03:31:58.201 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:31:58.204 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:31:58.207 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:31:58.210 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:31:58.229 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T03:31:58.236 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T03:31:58.244 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T03:31:58.244 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:31:58.263 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:31:58.265 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:31:58.270 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:31:58.358 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T03:31:58.366 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T03:31:58.366 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:31:58.369 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:31:58.372 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:31:58.374 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:31:58.406 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:58.410 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:58.418 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:58.423 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:58.425 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:58.426 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:58.437 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:58.438 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:58.439 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T03:31:58.446 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T03:31:58.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:31:58.468 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:31:58.472 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:31:58.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:31:58.502 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T03:31:58.512 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T03:31:58.512 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:31:58.538 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:31:58.541 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:31:58.543 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:31:58.561 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:31:58.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:31:58.685 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:31:58.768 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:31:58.812 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:31:58.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:31:58.880 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:31:58.922 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:31:58.992 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:31:59.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:31:59.079 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:31:59.079 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:31:59.145 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:31:59.191 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:31:59.251 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:31:59.293 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:31:59.338 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:31:59.338 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:31:59.405 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:31:59.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:31:59.776 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:31:59.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:31:59.865 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:31:59.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:31:59.933 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2024-09-16T03:31:59.935 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-09-16T03:31:59.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-09-16T03:31:59.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-09-16T03:31:59.975 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:31:59.989 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:31:59.993 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:32:00.017 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:32:00.359 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T03:32:00.366 INFO:teuthology.task.internal:Starting timer... 2024-09-16T03:32:00.367 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T03:32:00.377 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T03:32:00.386 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T03:32:00.387 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2024-09-16T03:32:00.461 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: 2024-09-16T03:32:00.463 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-09-16T03:32:00.515 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-09-16T03:32:00.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-09-16T03:32:00.566 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-09-16T03:32:00.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-09-16T03:32:00.630 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-09-16T03:32:00.631 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T03:32:00.632 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2024-09-16T03:32:00.663 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2024-09-16T03:32:00.664 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-09-16T03:32:00.690 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-09-16T03:32:00.690 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-09-16T03:32:00.717 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-09-16T03:32:00.717 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-09-16T03:32:00.744 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-09-16T03:32:00.745 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi076.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2024-09-16T03:32:00.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:32:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:32:00.774 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:32:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:32:00.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:32:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:32:00.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:32:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:32:00.864 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T03:32:00.864 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:32:00.898 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:32:00.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:32:00.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:32:01.020 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T03:32:01.030 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T03:32:01.046 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T03:32:01.047 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi076.front.sepia.ceph.com,smithi103.front.sepia.ceph.com,smithi119.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T03:35:53.157 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2024-09-16T03:35:53.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T03:35:53.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:35:53.237 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-09-16T03:35:53.313 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T03:35:53.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T03:35:53.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:35:53.392 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-09-16T03:35:53.485 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T03:35:53.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-16T03:35:53.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:35:53.565 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-09-16T03:35:53.641 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-16T03:35:53.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-16T03:35:53.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:35:53.717 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-09-16T03:35:53.795 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-16T03:35:53.796 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T03:35:53.806 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T03:35:53.806 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:35:53.807 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:35:53.809 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:35:53.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:35:53.812 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:35:53.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:35:53.814 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:35:53.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:35:53.841 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T03:35:53.843 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T03:35:53.845 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T03:35:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T03:35:53.858 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T03:35:53.861 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T03:35:53.868 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T03:35:53.884 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T03:35:53.885 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-09-16T03:35:53.887 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2024-09-16T03:35:53.888 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-09-16T03:35:53.896 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-09-16T03:35:53.900 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-09-16T03:35:53.901 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2024-09-16T03:35:53.911 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-09-16T03:35:53.911 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T03:35:53.914 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T03:35:53.918 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T03:35:53.924 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-09-16T03:35:53.925 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T03:35:53.929 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T03:35:53.935 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T03:35:53.937 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T03:35:53.952 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T03:35:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:35:53.985 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:35:53.985 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-09-16T03:35:53.985 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.985 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.985 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.985 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:35:53.990 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:35:53.991 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-09-16T03:35:53.991 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.991 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.991 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.991 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.995 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:35:53.999 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:35:53.999 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-09-16T03:35:53.999 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.999 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.999 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:53.999 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:35:54.012 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:35:54.013 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-09-16T03:35:54.013 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:54.013 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:54.013 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:54.013 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:35:54.014 INFO:teuthology.run_tasks:Running task install... 2024-09-16T03:35:54.022 DEBUG:teuthology.task.install:project ceph 2024-09-16T03:35:54.022 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T03:35:54.022 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T03:35:54.022 INFO:teuthology.task.install:Using flavor: default 2024-09-16T03:35:54.027 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T03:35:54.027 INFO:teuthology.task.install:extra packages: [] 2024-09-16T03:35:54.028 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T03:35:54.028 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:35:54.030 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T03:35:54.031 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:35:54.031 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T03:35:54.032 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:35:54.033 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T03:35:54.033 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:35:54.179 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T03:35:54.179 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T03:35:54.186 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T03:35:54.186 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T03:35:54.193 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T03:35:54.194 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T03:35:54.197 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T03:35:54.197 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T03:35:54.291 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T03:35:54.292 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:35:54.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T03:35:54.295 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T03:35:54.295 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:35:54.295 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T03:35:54.303 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T03:35:54.303 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:35:54.303 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T03:35:54.313 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T03:35:54.313 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:35:54.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T03:35:54.318 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T03:35:54.319 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T03:35:54.322 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T03:35:54.322 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T03:35:54.333 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T03:35:54.334 DEBUG:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T03:35:54.340 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T03:35:54.340 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T03:35:54.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T03:35:54.387 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T03:35:54.402 DEBUG:teuthology.orchestra.run.smithi076:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T03:35:54.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T03:35:54.459 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T03:35:54.462 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T03:35:54.482 DEBUG:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T03:35:54.484 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T03:35:54.526 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-09-16T03:35:54.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-09-16T03:35:54.529 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-09-16T03:35:54.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-09-16T03:35:54.550 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-09-16T03:35:54.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-09-16T03:35:54.552 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2024-09-16T03:35:54.553 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum clean all 2024-09-16T03:35:54.763 INFO:teuthology.orchestra.run.smithi103.stdout:45 files removed 2024-09-16T03:35:54.764 INFO:teuthology.orchestra.run.smithi119.stdout:45 files removed 2024-09-16T03:35:54.779 INFO:teuthology.orchestra.run.smithi121.stdout:45 files removed 2024-09-16T03:35:54.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw 2024-09-16T03:35:54.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw 2024-09-16T03:35:54.793 INFO:teuthology.orchestra.run.smithi076.stdout:45 files removed 2024-09-16T03:35:54.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-09-16T03:35:54.821 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw 2024-09-16T03:35:55.342 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 311 kB/s | 83 kB 00:00 2024-09-16T03:35:55.343 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 308 kB/s | 83 kB 00:00 2024-09-16T03:35:55.367 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 304 kB/s | 83 kB 00:00 2024-09-16T03:35:55.378 INFO:teuthology.orchestra.run.smithi076.stdout:ceph packages for x86_64 303 kB/s | 83 kB 00:00 2024-09-16T03:35:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T03:35:55.628 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T03:35:55.662 INFO:teuthology.orchestra.run.smithi076.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T03:35:55.668 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 46 kB/s | 12 kB 00:00 2024-09-16T03:35:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-16T03:35:55.903 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-16T03:35:55.921 INFO:teuthology.orchestra.run.smithi076.stdout:ceph source packages 8.5 kB/s | 1.9 kB 00:00 2024-09-16T03:35:55.935 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-16T03:35:56.361 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-09-16T03:35:56.663 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-16T03:35:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-16T03:35:57.701 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-09-16T03:35:59.280 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-16T03:35:59.642 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-16T03:36:00.028 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 20 MB 00:02 2024-09-16T03:36:00.477 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-16T03:36:03.380 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 20 MB/s | 6.5 MB 00:00 2024-09-16T03:36:04.895 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-16T03:36:05.148 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 19 kB 00:00 2024-09-16T03:36:05.208 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-16T03:36:05.313 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.5 MB 00:01 2024-09-16T03:36:06.277 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:01 2024-09-16T03:36:06.579 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 19 kB 00:00 2024-09-16T03:36:06.733 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 119 kB/s | 19 kB 00:00 2024-09-16T03:36:06.864 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 103 kB/s | 19 kB 00:00 2024-09-16T03:36:08.166 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-16T03:36:08.253 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-16T03:36:08.326 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-16T03:36:11.683 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T03:36:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T03:36:13.709 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T03:36:13.724 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T03:36:14.051 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T03:36:14.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T03:36:14.060 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T03:36:14.061 INFO:teuthology.orchestra.run.smithi119.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T03:36:14.062 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T03:36:14.062 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-09-16T03:36:14.062 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T03:36:14.062 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:14.062 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:36:14.062 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:36:14.062 INFO:teuthology.orchestra.run.smithi119.stdout:Install 30 Packages 2024-09-16T03:36:14.062 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 2 Packages 2024-09-16T03:36:14.062 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:14.063 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 60 M 2024-09-16T03:36:14.063 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:36:15.101 INFO:teuthology.orchestra.run.smithi119.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 9.0 MB/s | 5.4 MB 00:00 2024-09-16T03:36:15.226 INFO:teuthology.orchestra.run.smithi119.stdout:(2/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 15 MB/s | 11 MB 00:00 2024-09-16T03:36:15.251 INFO:teuthology.orchestra.run.smithi119.stdout:(3/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 168 kB/s | 25 kB 00:00 2024-09-16T03:36:15.293 INFO:teuthology.orchestra.run.smithi119.stdout:(4/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 11 MB/s | 751 kB 00:00 2024-09-16T03:36:15.319 INFO:teuthology.orchestra.run.smithi119.stdout:(5/32): libradosstriper1-19.1.1-326.g8921c8ef.e 7.4 MB/s | 506 kB 00:00 2024-09-16T03:36:15.385 INFO:teuthology.orchestra.run.smithi119.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 677 kB/s | 45 kB 00:00 2024-09-16T03:36:15.478 INFO:teuthology.orchestra.run.smithi119.stdout:(7/32): python3-ceph-common-19.1.1-326.g8921c8e 1.5 MB/s | 136 kB 00:00 2024-09-16T03:36:15.553 INFO:teuthology.orchestra.run.smithi119.stdout:(8/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_64 21 MB/s | 5.3 MB 00:00 2024-09-16T03:36:15.578 INFO:teuthology.orchestra.run.smithi119.stdout:(9/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 1.6 MB/s | 161 kB 00:00 2024-09-16T03:36:15.603 INFO:teuthology.orchestra.run.smithi119.stdout:(10/32): python3-rados-19.1.1-326.g8921c8ef.el9 6.2 MB/s | 320 kB 00:00 2024-09-16T03:36:15.628 INFO:teuthology.orchestra.run.smithi119.stdout:(11/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 5.9 MB/s | 300 kB 00:00 2024-09-16T03:36:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-16T03:36:15.863 INFO:teuthology.orchestra.run.smithi119.stdout:(13/32): ceph-common-19.1.1-326.g8921c8ef.el9.x 16 MB/s | 21 MB 00:01 2024-09-16T03:36:15.938 INFO:teuthology.orchestra.run.smithi119.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 346 kB/s | 107 kB 00:00 2024-09-16T03:36:15.938 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:36:15.954 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:36:15.954 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-16T03:36:15.954 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:36:15.954 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:36:15.954 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T03:36:15.955 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T03:36:15.956 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:36:15.957 INFO:teuthology.orchestra.run.smithi121.stdout:Install 30 Packages 2024-09-16T03:36:15.958 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-09-16T03:36:15.958 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:15.958 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 60 M 2024-09-16T03:36:15.958 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:36:15.963 INFO:teuthology.orchestra.run.smithi119.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-09-16T03:36:15.997 INFO:teuthology.orchestra.run.smithi119.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 467 kB/s | 160 kB 00:00 2024-09-16T03:36:16.039 INFO:teuthology.orchestra.run.smithi119.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-16T03:36:16.051 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:36:16.056 INFO:teuthology.orchestra.run.smithi119.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 720 kB/s | 42 kB 00:00 2024-09-16T03:36:16.064 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:36:16.064 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T03:36:16.064 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:36:16.064 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:36:16.064 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T03:36:16.064 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2024-09-16T03:36:16.064 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T03:36:16.064 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T03:36:16.064 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T03:36:16.065 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T03:36:16.066 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T03:36:16.067 INFO:teuthology.orchestra.run.smithi076.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T03:36:16.067 INFO:teuthology.orchestra.run.smithi076.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T03:36:16.067 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2024-09-16T03:36:16.067 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T03:36:16.067 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:16.067 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:36:16.067 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:36:16.067 INFO:teuthology.orchestra.run.smithi076.stdout:Install 30 Packages 2024-09-16T03:36:16.067 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 2 Packages 2024-09-16T03:36:16.067 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:16.068 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 60 M 2024-09-16T03:36:16.068 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:36:16.081 INFO:teuthology.orchestra.run.smithi119.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-16T03:36:16.083 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:36:16.094 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T03:36:16.095 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T03:36:16.096 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T03:36:16.097 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T03:36:16.098 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-09-16T03:36:16.098 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T03:36:16.098 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:16.098 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:36:16.098 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:36:16.098 INFO:teuthology.orchestra.run.smithi103.stdout:Install 30 Packages 2024-09-16T03:36:16.098 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 2 Packages 2024-09-16T03:36:16.098 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:16.099 INFO:teuthology.orchestra.run.smithi119.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 856 kB/s | 36 kB 00:00 2024-09-16T03:36:16.099 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 60 M 2024-09-16T03:36:16.099 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:36:16.123 INFO:teuthology.orchestra.run.smithi119.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-16T03:36:16.140 INFO:teuthology.orchestra.run.smithi119.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 806 kB/s | 47 kB 00:00 2024-09-16T03:36:16.199 INFO:teuthology.orchestra.run.smithi119.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-16T03:36:16.216 INFO:teuthology.orchestra.run.smithi119.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-09-16T03:36:16.233 INFO:teuthology.orchestra.run.smithi119.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 335 kB/s | 26 kB 00:00 2024-09-16T03:36:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-16T03:36:16.300 INFO:teuthology.orchestra.run.smithi119.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T03:36:16.359 INFO:teuthology.orchestra.run.smithi119.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-09-16T03:36:16.417 INFO:teuthology.orchestra.run.smithi119.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-16T03:36:16.509 INFO:teuthology.orchestra.run.smithi119.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-16T03:36:16.634 INFO:teuthology.orchestra.run.smithi119.stdout:(31/32): librados2-19.1.1-326.g8921c8ef.el9.x86 12 MB/s | 3.4 MB 00:00 2024-09-16T03:36:16.818 INFO:teuthology.orchestra.run.smithi119.stdout:(32/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 7.9 MB/s | 3.2 MB 00:00 2024-09-16T03:36:16.823 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:16.824 INFO:teuthology.orchestra.run.smithi119.stdout:Total 22 MB/s | 60 MB 00:02 2024-09-16T03:36:16.836 INFO:teuthology.orchestra.run.smithi076.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 8.2 MB/s | 5.4 MB 00:00 2024-09-16T03:36:16.902 INFO:teuthology.orchestra.run.smithi076.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 378 kB/s | 25 kB 00:00 2024-09-16T03:36:16.932 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:36:16.967 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:36:16.967 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:36:17.011 INFO:teuthology.orchestra.run.smithi076.stdout:(3/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 6.8 MB/s | 751 kB 00:00 2024-09-16T03:36:17.120 INFO:teuthology.orchestra.run.smithi076.stdout:(4/32): libradosstriper1-19.1.1-326.g8921c8ef.e 4.6 MB/s | 506 kB 00:00 2024-09-16T03:36:17.278 INFO:teuthology.orchestra.run.smithi076.stdout:(5/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 9.6 MB/s | 11 MB 00:01 2024-09-16T03:36:17.346 INFO:teuthology.orchestra.run.smithi076.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 676 kB/s | 45 kB 00:00 2024-09-16T03:36:17.437 INFO:teuthology.orchestra.run.smithi076.stdout:(7/32): python3-ceph-common-19.1.1-326.g8921c8e 1.5 MB/s | 136 kB 00:00 2024-09-16T03:36:17.641 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:36:17.641 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:36:17.654 INFO:teuthology.orchestra.run.smithi076.stdout:(8/32): ceph-common-19.1.1-326.g8921c8ef.el9.x8 14 MB/s | 21 MB 00:01 2024-09-16T03:36:17.681 INFO:teuthology.orchestra.run.smithi076.stdout:(9/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 667 kB/s | 161 kB 00:00 2024-09-16T03:36:17.722 INFO:teuthology.orchestra.run.smithi076.stdout:(10/32): python3-rados-19.1.1-326.g8921c8ef.el9 4.7 MB/s | 320 kB 00:00 2024-09-16T03:36:17.797 INFO:teuthology.orchestra.run.smithi076.stdout:(11/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_6 7.9 MB/s | 5.3 MB 00:00 2024-09-16T03:36:17.822 INFO:teuthology.orchestra.run.smithi076.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 991 kB/s | 99 kB 00:00 2024-09-16T03:36:17.847 INFO:teuthology.orchestra.run.smithi076.stdout:(13/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 1.8 MB/s | 300 kB 00:00 2024-09-16T03:36:17.956 INFO:teuthology.orchestra.run.smithi076.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 673 kB/s | 107 kB 00:00 2024-09-16T03:36:17.972 INFO:teuthology.orchestra.run.smithi121.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 3.5 MB/s | 5.4 MB 00:01 2024-09-16T03:36:17.981 INFO:teuthology.orchestra.run.smithi076.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 339 kB/s | 45 kB 00:00 2024-09-16T03:36:18.007 INFO:teuthology.orchestra.run.smithi076.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 869 kB/s | 160 kB 00:00 2024-09-16T03:36:18.022 INFO:teuthology.orchestra.run.smithi121.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 506 kB/s | 25 kB 00:00 2024-09-16T03:36:18.042 INFO:teuthology.orchestra.run.smithi103.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 3.1 MB/s | 5.4 MB 00:01 2024-09-16T03:36:18.082 INFO:teuthology.orchestra.run.smithi076.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 564 kB/s | 42 kB 00:00 2024-09-16T03:36:18.092 INFO:teuthology.orchestra.run.smithi103.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 504 kB/s | 25 kB 00:00 2024-09-16T03:36:18.131 INFO:teuthology.orchestra.run.smithi121.stdout:(3/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 6.8 MB/s | 751 kB 00:00 2024-09-16T03:36:18.132 INFO:teuthology.orchestra.run.smithi076.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-16T03:36:18.166 INFO:teuthology.orchestra.run.smithi076.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-16T03:36:18.191 INFO:teuthology.orchestra.run.smithi076.stdout:(20/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 662 kB 00:00 2024-09-16T03:36:18.209 INFO:teuthology.orchestra.run.smithi103.stdout:(3/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 6.3 MB/s | 751 kB 00:00 2024-09-16T03:36:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:(4/32): libradosstriper1-19.1.1-326.g8921c8ef.e 5.9 MB/s | 506 kB 00:00 2024-09-16T03:36:18.216 INFO:teuthology.orchestra.run.smithi076.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 428 kB/s | 36 kB 00:00 2024-09-16T03:36:18.241 INFO:teuthology.orchestra.run.smithi076.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-09-16T03:36:18.308 INFO:teuthology.orchestra.run.smithi076.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-16T03:36:18.326 INFO:teuthology.orchestra.run.smithi103.stdout:(4/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 5.2 MB/s | 11 MB 00:02 2024-09-16T03:36:18.334 INFO:teuthology.orchestra.run.smithi076.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 336 kB/s | 26 kB 00:00 2024-09-16T03:36:18.352 INFO:teuthology.orchestra.run.smithi103.stdout:(5/32): libradosstriper1-19.1.1-326.g8921c8ef.e 3.5 MB/s | 506 kB 00:00 2024-09-16T03:36:18.359 INFO:teuthology.orchestra.run.smithi076.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-09-16T03:36:18.373 INFO:teuthology.orchestra.run.smithi121.stdout:(5/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 5.4 MB/s | 11 MB 00:01 2024-09-16T03:36:18.384 INFO:teuthology.orchestra.run.smithi076.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T03:36:18.410 INFO:teuthology.orchestra.run.smithi103.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 772 kB/s | 45 kB 00:00 2024-09-16T03:36:18.423 INFO:teuthology.orchestra.run.smithi121.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 907 kB/s | 45 kB 00:00 2024-09-16T03:36:18.434 INFO:teuthology.orchestra.run.smithi076.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-16T03:36:18.460 INFO:teuthology.orchestra.run.smithi103.stdout:(7/32): python3-ceph-common-19.1.1-326.g8921c8e 2.7 MB/s | 136 kB 00:00 2024-09-16T03:36:18.476 INFO:teuthology.orchestra.run.smithi076.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-16T03:36:18.482 INFO:teuthology.orchestra.run.smithi121.stdout:(7/32): python3-ceph-common-19.1.1-326.g8921c8e 2.3 MB/s | 136 kB 00:00 2024-09-16T03:36:18.519 INFO:teuthology.orchestra.run.smithi103.stdout:(8/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 2.7 MB/s | 161 kB 00:00 2024-09-16T03:36:18.549 INFO:teuthology.orchestra.run.smithi121.stdout:(8/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 2.4 MB/s | 161 kB 00:00 2024-09-16T03:36:18.586 INFO:teuthology.orchestra.run.smithi103.stdout:(9/32): python3-rados-19.1.1-326.g8921c8ef.el9. 4.7 MB/s | 320 kB 00:00 2024-09-16T03:36:18.666 INFO:teuthology.orchestra.run.smithi121.stdout:(9/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_64 12 MB/s | 5.3 MB 00:00 2024-09-16T03:36:18.670 INFO:teuthology.orchestra.run.smithi103.stdout:(10/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_6 16 MB/s | 5.3 MB 00:00 2024-09-16T03:36:18.685 INFO:teuthology.orchestra.run.smithi076.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-09-16T03:36:18.699 INFO:teuthology.orchestra.run.smithi121.stdout:(10/32): python3-rados-19.1.1-326.g8921c8ef.el9 2.1 MB/s | 320 kB 00:00 2024-09-16T03:36:18.703 INFO:teuthology.orchestra.run.smithi103.stdout:(11/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 2.5 MB/s | 300 kB 00:00 2024-09-16T03:36:18.729 INFO:teuthology.orchestra.run.smithi103.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.7 MB/s | 99 kB 00:00 2024-09-16T03:36:18.733 INFO:teuthology.orchestra.run.smithi121.stdout:(11/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 4.4 MB/s | 300 kB 00:00 2024-09-16T03:36:18.758 INFO:teuthology.orchestra.run.smithi121.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.7 MB/s | 99 kB 00:00 2024-09-16T03:36:18.827 INFO:teuthology.orchestra.run.smithi076.stdout:(30/32): librados2-19.1.1-326.g8921c8ef.el9.x86 9.8 MB/s | 3.4 MB 00:00 2024-09-16T03:36:18.829 INFO:teuthology.orchestra.run.smithi103.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 851 kB/s | 107 kB 00:00 2024-09-16T03:36:18.854 INFO:teuthology.orchestra.run.smithi103.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-16T03:36:18.876 INFO:teuthology.orchestra.run.smithi121.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 754 kB/s | 107 kB 00:00 2024-09-16T03:36:18.880 INFO:teuthology.orchestra.run.smithi103.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 902 kB/s | 45 kB 00:00 2024-09-16T03:36:18.894 INFO:teuthology.orchestra.run.smithi076.stdout:(31/32): libarrow-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 4.4 MB 00:00 2024-09-16T03:36:18.909 INFO:teuthology.orchestra.run.smithi121.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-16T03:36:18.935 INFO:teuthology.orchestra.run.smithi121.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 770 kB/s | 45 kB 00:00 2024-09-16T03:36:18.947 INFO:teuthology.orchestra.run.smithi103.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-09-16T03:36:18.953 INFO:teuthology.orchestra.run.smithi076.stdout:(32/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-16T03:36:18.958 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:18.958 INFO:teuthology.orchestra.run.smithi076.stdout:Total 21 MB/s | 60 MB 00:02 2024-09-16T03:36:18.972 INFO:teuthology.orchestra.run.smithi103.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-16T03:36:18.989 INFO:teuthology.orchestra.run.smithi103.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-16T03:36:19.016 INFO:teuthology.orchestra.run.smithi103.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 221 kB 00:00 2024-09-16T03:36:19.018 INFO:teuthology.orchestra.run.smithi121.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-09-16T03:36:19.031 INFO:teuthology.orchestra.run.smithi103.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 854 kB/s | 36 kB 00:00 2024-09-16T03:36:19.043 INFO:teuthology.orchestra.run.smithi121.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-16T03:36:19.048 INFO:teuthology.orchestra.run.smithi103.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-16T03:36:19.058 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:36:19.069 INFO:teuthology.orchestra.run.smithi121.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 839 kB/s | 42 kB 00:00 2024-09-16T03:36:19.096 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:36:19.096 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:36:19.098 INFO:teuthology.orchestra.run.smithi103.stdout:(22/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.5 MB/s | 308 kB 00:00 2024-09-16T03:36:19.141 INFO:teuthology.orchestra.run.smithi103.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 648 kB/s | 26 kB 00:00 2024-09-16T03:36:19.155 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:36:19.166 INFO:teuthology.orchestra.run.smithi103.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T03:36:19.200 INFO:teuthology.orchestra.run.smithi103.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-16T03:36:19.242 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T03:36:19.267 INFO:teuthology.orchestra.run.smithi103.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-16T03:36:19.269 INFO:teuthology.orchestra.run.smithi121.stdout:(19/32): ceph-common-19.1.1-326.g8921c8ef.el9.x 7.4 MB/s | 21 MB 00:02 2024-09-16T03:36:19.272 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T03:36:19.294 INFO:teuthology.orchestra.run.smithi121.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 159 kB/s | 36 kB 00:00 2024-09-16T03:36:19.298 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T03:36:19.300 INFO:teuthology.orchestra.run.smithi103.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T03:36:19.320 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:36:19.327 INFO:teuthology.orchestra.run.smithi121.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 777 kB/s | 221 kB 00:00 2024-09-16T03:36:19.347 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:36:19.386 INFO:teuthology.orchestra.run.smithi121.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 406 kB/s | 47 kB 00:00 2024-09-16T03:36:19.420 INFO:teuthology.orchestra.run.smithi121.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 MB/s | 308 kB 00:00 2024-09-16T03:36:19.434 INFO:teuthology.orchestra.run.smithi103.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-16T03:36:19.445 INFO:teuthology.orchestra.run.smithi121.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-09-16T03:36:19.470 INFO:teuthology.orchestra.run.smithi121.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.0 MB/s | 49 kB 00:00 2024-09-16T03:36:19.476 INFO:teuthology.orchestra.run.smithi103.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-09-16T03:36:19.496 INFO:teuthology.orchestra.run.smithi121.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T03:36:19.529 INFO:teuthology.orchestra.run.smithi121.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T03:36:19.558 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:36:19.588 INFO:teuthology.orchestra.run.smithi121.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-16T03:36:19.609 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:36:19.672 INFO:teuthology.orchestra.run.smithi121.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-16T03:36:19.747 INFO:teuthology.orchestra.run.smithi121.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-16T03:36:19.761 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:36:19.762 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:36:19.986 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T03:36:20.058 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:20.132 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:20.228 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T03:36:20.251 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:36:20.260 INFO:teuthology.orchestra.run.smithi103.stdout:(30/32): librados2-19.1.1-326.g8921c8ef.el9.x86 4.1 MB/s | 3.4 MB 00:00 2024-09-16T03:36:20.272 INFO:teuthology.orchestra.run.smithi121.stdout:(31/32): librados2-19.1.1-326.g8921c8ef.el9.x86 5.0 MB/s | 3.4 MB 00:00 2024-09-16T03:36:20.314 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:36:20.318 INFO:teuthology.orchestra.run.smithi103.stdout:(31/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 3.7 MB/s | 3.2 MB 00:00 2024-09-16T03:36:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T03:36:20.402 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T03:36:20.470 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:36:20.489 INFO:teuthology.orchestra.run.smithi121.stdout:(32/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 3.9 MB/s | 3.2 MB 00:00 2024-09-16T03:36:20.494 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:20.494 INFO:teuthology.orchestra.run.smithi121.stdout:Total 13 MB/s | 60 MB 00:04 2024-09-16T03:36:20.495 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:20.559 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:20.594 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T03:36:20.601 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:36:20.637 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:36:20.637 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:36:20.702 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T03:36:20.736 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T03:36:20.777 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T03:36:20.818 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T03:36:20.855 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T03:36:20.897 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T03:36:21.206 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T03:36:21.240 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:36:21.251 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:36:21.277 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:36:21.277 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:36:21.320 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T03:36:21.340 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:36:21.350 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T03:36:21.373 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T03:36:21.374 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T03:36:21.397 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:36:21.417 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:36:21.418 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T03:36:21.617 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:36:21.669 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:36:21.895 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T03:36:21.920 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:36:21.991 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:36:22.043 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T03:36:22.069 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T03:36:22.098 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T03:36:22.118 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:22.210 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:22.302 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T03:36:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:36:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:36:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T03:36:22.481 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T03:36:22.558 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:36:22.580 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:22.684 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T03:36:22.758 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:36:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T03:36:22.838 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T03:36:22.839 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T03:36:22.869 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T03:36:22.879 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T03:36:22.894 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T03:36:22.911 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T03:36:22.917 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:36:22.941 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T03:36:22.941 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:36:22.974 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T03:36:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:36:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:36:23.286 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T03:36:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:36:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:36:23.450 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T03:36:23.493 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T03:36:23.509 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T03:36:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:23.712 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:23.803 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T03:36:23.830 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:36:23.885 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:36:23.914 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T03:36:23.973 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T03:36:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T03:36:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:36:24.044 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:36:24.059 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:36:24.066 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:24.128 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T03:36:24.130 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:24.158 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T03:36:24.165 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T03:36:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T03:36:24.316 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T03:36:24.357 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T03:36:24.397 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T03:36:24.418 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:24.426 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T03:36:24.452 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:24.463 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T03:36:24.535 INFO:teuthology.orchestra.run.smithi103.stdout:(32/32): ceph-common-19.1.1-326.g8921c8ef.el9.x 2.6 MB/s | 21 MB 00:08 2024-09-16T03:36:24.541 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:24.541 INFO:teuthology.orchestra.run.smithi103.stdout:Total 7.2 MB/s | 60 MB 00:08 2024-09-16T03:36:24.646 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:36:24.683 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:36:24.683 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:36:24.776 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T03:36:24.822 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:36:24.912 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:36:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T03:36:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T03:36:25.005 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:25.048 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:36:25.132 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:36:25.132 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T03:36:25.132 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T03:36:25.133 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:25.166 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:25.357 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:36:25.358 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:36:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T03:36:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:36:25.573 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:36:25.647 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T03:36:25.677 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T03:36:26.669 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:26.704 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:26.868 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:36:26.959 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T03:36:26.990 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T03:36:27.016 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T03:36:27.038 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:36:27.061 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:36:27.249 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:36:27.274 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:27.310 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:36:27.316 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:36:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:36:27.413 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T03:36:27.413 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T03:36:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T03:36:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:27.827 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:27.916 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T03:36:27.946 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:36:27.954 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:27.999 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:27.999 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:36:28.036 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T03:36:28.084 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T03:36:28.159 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:36:28.189 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:28.236 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:28.271 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T03:36:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T03:36:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T03:36:28.454 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T03:36:28.487 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T03:36:28.516 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T03:36:28.549 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T03:36:28.570 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:28.611 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:36:28.687 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:36:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T03:36:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T03:36:28.687 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:28.720 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:28.875 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T03:36:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:36:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:36:29.042 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T03:36:29.078 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T03:36:29.570 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T03:36:29.606 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:36:29.661 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:36:29.739 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T03:36:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T03:36:32.081 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:32.701 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:36:32.735 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:36:32.818 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:36:32.818 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T03:36:32.818 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T03:36:32.818 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:35.355 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:35.355 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-09-16T03:36:35.355 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-09-16T03:36:35.355 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-09-16T03:36:35.356 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-09-16T03:36:35.356 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-09-16T03:36:35.356 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-09-16T03:36:35.356 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-09-16T03:36:35.356 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:36.332 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:36:36.358 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:36:36.358 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:36:36.358 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T03:36:36.358 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:36:36.358 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:36:36.358 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:36.359 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:36:36.452 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:36:36.452 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:37.332 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:37.332 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /sys 2024-09-16T03:36:37.332 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /proc 2024-09-16T03:36:37.333 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /mnt 2024-09-16T03:36:37.333 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /var/tmp 2024-09-16T03:36:37.333 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /home 2024-09-16T03:36:37.333 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /root 2024-09-16T03:36:37.333 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /tmp 2024-09-16T03:36:37.333 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:37.687 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:37.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T03:36:37.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T03:36:37.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T03:36:37.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T03:36:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T03:36:37.689 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T03:36:37.690 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T03:36:37.690 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:37.690 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T03:36:37.690 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T03:36:38.328 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:36:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:36:38.354 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:36:38.355 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T03:36:38.356 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:36:38.356 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:36:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:36:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:36:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:38.733 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:38.733 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:38.733 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T03:36:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T03:36:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T03:36:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T03:36:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T03:36:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T03:36:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T03:36:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:38.736 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:36:38.829 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:38.829 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-09-16T03:36:38.829 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-09-16T03:36:38.829 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-09-16T03:36:38.829 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-09-16T03:36:38.830 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-09-16T03:36:38.830 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-09-16T03:36:38.830 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-09-16T03:36:38.830 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:39.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-test 2024-09-16T03:36:39.660 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:36:39.684 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:39.684 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T03:36:39.684 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:39.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T03:36:39.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T03:36:39.687 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T03:36:39.687 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T03:36:39.687 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T03:36:39.688 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:39.688 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T03:36:39.688 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T03:36:39.788 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:36:39.793 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:36:39.794 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:36:39.794 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-16T03:36:39.794 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout:Install 4 Packages 2024-09-16T03:36:39.795 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:39.796 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 49 M 2024-09-16T03:36:39.796 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 209 M 2024-09-16T03:36:39.796 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:36:39.813 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:36:39.813 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:36:39.813 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T03:36:39.813 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:36:39.813 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:36:39.813 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:39.815 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:36:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:36:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-16T03:36:40.148 INFO:teuthology.orchestra.run.smithi119.stdout:(2/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 700 kB/s | 163 kB 00:00 2024-09-16T03:36:40.189 INFO:teuthology.orchestra.run.smithi119.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 972 kB/s | 64 kB 00:00 2024-09-16T03:36:40.737 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:40.737 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:40.737 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2024-09-16T03:36:40.737 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.737 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.737 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:36:40.737 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.738 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T03:36:40.739 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T03:36:40.740 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T03:36:40.740 INFO:teuthology.orchestra.run.smithi076.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T03:36:40.740 INFO:teuthology.orchestra.run.smithi076.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T03:36:40.740 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:40.740 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:36:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T03:36:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T03:36:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T03:36:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T03:36:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T03:36:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T03:36:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T03:36:41.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-test 2024-09-16T03:36:41.599 INFO:teuthology.orchestra.run.smithi119.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 29 MB/s | 49 MB 00:01 2024-09-16T03:36:41.601 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:41.601 INFO:teuthology.orchestra.run.smithi119.stdout:Total 27 MB/s | 49 MB 00:01 2024-09-16T03:36:41.629 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:36:41.648 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:36:41.648 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:36:41.735 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:36:41.866 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:36:41.867 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:36:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T03:36:41.867 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:36:41.867 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:36:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T03:36:41.867 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T03:36:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T03:36:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T03:36:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T03:36:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:41.874 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:36:41.874 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:36:41.874 INFO:teuthology.orchestra.run.smithi076.stdout:Install 4 Packages 2024-09-16T03:36:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:41.875 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 49 M 2024-09-16T03:36:41.875 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 209 M 2024-09-16T03:36:41.875 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:36:42.032 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:36:42.032 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:36:42.125 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:42.125 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:42.125 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T03:36:42.126 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.127 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:42.128 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:36:42.135 INFO:teuthology.orchestra.run.smithi076.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-16T03:36:42.161 INFO:teuthology.orchestra.run.smithi076.stdout:(2/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 664 kB/s | 163 kB 00:00 2024-09-16T03:36:42.186 INFO:teuthology.orchestra.run.smithi076.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-16T03:36:42.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-09-16T03:36:42.556 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:36:42.638 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T03:36:42.681 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T03:36:42.709 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:36:42.709 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:42.709 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-09-16T03:36:42.709 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-09-16T03:36:42.710 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-09-16T03:36:42.710 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-09-16T03:36:42.710 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-09-16T03:36:42.710 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-09-16T03:36:42.710 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-09-16T03:36:42.710 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:43.008 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:30 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:36:43.144 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:36:43.145 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:36:43.145 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-16T03:36:43.145 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:36:43.145 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:36:43.145 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T03:36:43.145 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-16T03:36:43.145 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T03:36:43.145 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T03:36:43.146 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T03:36:43.146 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:43.146 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:36:43.146 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:36:43.146 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-09-16T03:36:43.146 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:43.146 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 49 M 2024-09-16T03:36:43.147 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 209 M 2024-09-16T03:36:43.147 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:36:43.389 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-16T03:36:43.423 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 643 kB/s | 163 kB 00:00 2024-09-16T03:36:43.448 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-16T03:36:43.594 INFO:teuthology.orchestra.run.smithi076.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 29 MB/s | 49 MB 00:01 2024-09-16T03:36:43.596 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:43.597 INFO:teuthology.orchestra.run.smithi076.stdout:Total 28 MB/s | 49 MB 00:01 2024-09-16T03:36:43.622 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:36:43.642 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:36:43.642 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:36:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:36:43.722 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:36:43.722 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:36:43.722 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T03:36:43.722 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:36:43.722 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:36:43.722 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:43.724 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:36:43.806 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:36:43.806 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:44.028 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:36:44.028 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:36:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:36:44.638 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T03:36:44.676 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T03:36:44.705 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:36:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T03:36:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T03:36:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T03:36:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T03:36:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T03:36:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T03:36:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T03:36:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T03:36:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T03:36:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:36:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T03:36:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T03:36:45.073 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 26 MB/s | 49 MB 00:01 2024-09-16T03:36:45.075 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:45.076 INFO:teuthology.orchestra.run.smithi121.stdout:Total 25 MB/s | 49 MB 00:01 2024-09-16T03:36:45.103 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:36:45.123 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:36:45.124 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:36:45.515 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:36:45.516 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:36:46.005 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:36:46.005 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-09-16T03:36:46.005 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.005 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T03:36:46.006 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T03:36:46.007 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.007 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T03:36:46.007 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.007 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T03:36:46.007 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T03:36:46.007 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T03:36:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T03:36:46.009 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:46.009 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:36:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:36:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T03:36:46.163 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T03:36:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:36:46.377 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test 2024-09-16T03:36:46.947 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:33 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:36:47.086 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:36:47.087 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:36:47.088 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-09-16T03:36:47.088 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:47.088 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 49 M 2024-09-16T03:36:47.088 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 209 M 2024-09-16T03:36:47.088 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:36:47.263 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-16T03:36:47.305 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T03:36:47.306 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:36:47.372 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 690 kB/s | 163 kB 00:00 2024-09-16T03:36:47.391 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:36:49.286 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:36:49.369 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:36:49.514 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 20 MB/s | 49 MB 00:02 2024-09-16T03:36:49.516 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:49.517 INFO:teuthology.orchestra.run.smithi103.stdout:Total 20 MB/s | 49 MB 00:02 2024-09-16T03:36:49.543 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:36:49.563 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:36:49.564 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:36:49.799 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:36:49.799 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T03:36:49.799 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T03:36:49.800 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T03:36:49.952 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:36:49.953 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:36:50.142 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T03:36:50.142 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:50.143 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:36:50.143 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:50.143 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:50.143 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T03:36:50.143 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T03:36:50.143 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:50.143 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:36:50.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph 2024-09-16T03:36:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:36:50.550 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T03:36:50.588 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T03:36:50.616 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:36:50.835 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:36:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:36:50.995 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:39 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:36:51.133 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T03:36:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T03:36:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T03:36:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T03:36:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T03:36:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T03:36:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T03:36:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T03:36:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T03:36:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T03:36:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T03:36:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T03:36:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T03:36:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T03:36:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T03:36:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T03:36:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T03:36:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T03:36:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T03:36:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T03:36:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T03:36:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T03:36:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T03:36:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:51.140 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:36:51.140 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:36:51.140 INFO:teuthology.orchestra.run.smithi119.stdout:Install 37 Packages 2024-09-16T03:36:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:51.142 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 30 M 2024-09-16T03:36:51.142 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 108 M 2024-09-16T03:36:51.142 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:36:51.799 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:36:51.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T03:36:51.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T03:36:51.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T03:36:52.149 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T03:36:52.149 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:52.149 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:36:52.149 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:52.149 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:52.149 INFO:teuthology.orchestra.run.smithi076.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T03:36:52.149 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T03:36:52.149 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:52.149 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:36:52.161 INFO:teuthology.orchestra.run.smithi119.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 51 kB/s | 6.5 kB 00:00 2024-09-16T03:36:52.387 INFO:teuthology.orchestra.run.smithi119.stdout:(2/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.8 MB/s | 2.4 MB 00:00 2024-09-16T03:36:52.420 INFO:teuthology.orchestra.run.smithi119.stdout:(3/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 3.9 MB/s | 1.5 MB 00:00 2024-09-16T03:36:52.433 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph 2024-09-16T03:36:52.496 INFO:teuthology.orchestra.run.smithi119.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 3.3 MB/s | 249 kB 00:00 2024-09-16T03:36:52.737 INFO:teuthology.orchestra.run.smithi119.stdout:(5/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 8.2 MB/s | 4.7 MB 00:00 2024-09-16T03:36:52.938 INFO:teuthology.orchestra.run.smithi119.stdout:(6/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 31 MB/s | 17 MB 00:00 2024-09-16T03:36:53.005 INFO:teuthology.orchestra.run.smithi119.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 498 kB/s | 253 kB 00:00 2024-09-16T03:36:53.019 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:36:53.138 INFO:teuthology.orchestra.run.smithi119.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.0 MB/s | 135 kB 00:00 2024-09-16T03:36:53.163 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:36:53.165 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:36:53.165 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T03:36:53.165 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:36:53.166 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:36:53.166 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T03:36:53.166 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T03:36:53.166 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T03:36:53.166 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T03:36:53.166 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T03:36:53.166 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T03:36:53.166 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T03:36:53.166 INFO:teuthology.orchestra.run.smithi076.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T03:36:53.166 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T03:36:53.167 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T03:36:53.168 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T03:36:53.169 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T03:36:53.169 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T03:36:53.169 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T03:36:53.169 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T03:36:53.169 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:53.169 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:36:53.169 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:36:53.169 INFO:teuthology.orchestra.run.smithi076.stdout:Install 37 Packages 2024-09-16T03:36:53.169 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:53.170 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 30 M 2024-09-16T03:36:53.170 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 108 M 2024-09-16T03:36:53.170 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:36:53.255 INFO:teuthology.orchestra.run.smithi119.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 336 kB/s | 106 kB 00:00 2024-09-16T03:36:53.281 INFO:teuthology.orchestra.run.smithi119.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 880 kB/s | 125 kB 00:00 2024-09-16T03:36:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:36:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T03:36:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T03:36:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T03:36:53.472 INFO:teuthology.orchestra.run.smithi119.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 990 kB/s | 215 kB 00:00 2024-09-16T03:36:53.498 INFO:teuthology.orchestra.run.smithi119.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 868 kB/s | 188 kB 00:00 2024-09-16T03:36:53.548 INFO:teuthology.orchestra.run.smithi119.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-16T03:36:53.598 INFO:teuthology.orchestra.run.smithi119.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-16T03:36:53.624 INFO:teuthology.orchestra.run.smithi119.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 277 kB/s | 35 kB 00:00 2024-09-16T03:36:53.649 INFO:teuthology.orchestra.run.smithi119.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T03:36:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T03:36:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:53.654 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:36:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T03:36:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T03:36:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:53.654 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:36:53.662 INFO:teuthology.orchestra.run.smithi076.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 46 kB/s | 6.5 kB 00:00 2024-09-16T03:36:53.699 INFO:teuthology.orchestra.run.smithi119.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 554 kB/s | 42 kB 00:00 2024-09-16T03:36:53.758 INFO:teuthology.orchestra.run.smithi119.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-16T03:36:53.783 INFO:teuthology.orchestra.run.smithi119.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-09-16T03:36:53.817 INFO:teuthology.orchestra.run.smithi119.stdout:(20/37): python3-cryptography-36.0.1-4.el9.x86_ 1.1 MB/s | 1.2 MB 00:01 2024-09-16T03:36:53.850 INFO:teuthology.orchestra.run.smithi119.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 471 kB/s | 43 kB 00:00 2024-09-16T03:36:53.854 INFO:teuthology.orchestra.run.smithi076.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 4.5 MB/s | 1.5 MB 00:00 2024-09-16T03:36:53.884 INFO:teuthology.orchestra.run.smithi119.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 450 kB/s | 11 kB 00:00 2024-09-16T03:36:53.904 INFO:teuthology.orchestra.run.smithi076.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.2 MB/s | 2.4 MB 00:00 2024-09-16T03:36:53.909 INFO:teuthology.orchestra.run.smithi119.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-16T03:36:53.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-09-16T03:36:53.934 INFO:teuthology.orchestra.run.smithi119.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 366 kB/s | 18 kB 00:00 2024-09-16T03:36:53.960 INFO:teuthology.orchestra.run.smithi119.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-16T03:36:53.977 INFO:teuthology.orchestra.run.smithi119.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 463 kB/s | 19 kB 00:00 2024-09-16T03:36:53.996 INFO:teuthology.orchestra.run.smithi076.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 2.7 MB/s | 249 kB 00:00 2024-09-16T03:36:54.023 INFO:teuthology.orchestra.run.smithi119.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 312 kB/s | 20 kB 00:00 2024-09-16T03:36:54.044 INFO:teuthology.orchestra.run.smithi119.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 694 kB/s | 46 kB 00:00 2024-09-16T03:36:54.069 INFO:teuthology.orchestra.run.smithi119.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.7 MB/s | 79 kB 00:00 2024-09-16T03:36:54.086 INFO:teuthology.orchestra.run.smithi119.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 985 kB/s | 16 kB 00:00 2024-09-16T03:36:54.112 INFO:teuthology.orchestra.run.smithi119.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-16T03:36:54.137 INFO:teuthology.orchestra.run.smithi119.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-16T03:36:54.155 INFO:teuthology.orchestra.run.smithi076.stdout:(5/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 9.6 MB/s | 4.7 MB 00:00 2024-09-16T03:36:54.162 INFO:teuthology.orchestra.run.smithi119.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-16T03:36:54.180 INFO:teuthology.orchestra.run.smithi076.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-16T03:36:54.188 INFO:teuthology.orchestra.run.smithi119.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-16T03:36:54.213 INFO:teuthology.orchestra.run.smithi119.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 810 kB/s | 20 kB 00:00 2024-09-16T03:36:54.238 INFO:teuthology.orchestra.run.smithi119.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T03:36:54.247 INFO:teuthology.orchestra.run.smithi076.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-09-16T03:36:54.263 INFO:teuthology.orchestra.run.smithi119.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-16T03:36:54.268 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:54.269 INFO:teuthology.orchestra.run.smithi119.stdout:Total 9.7 MB/s | 30 MB 00:03 2024-09-16T03:36:54.423 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:36:54.431 INFO:teuthology.orchestra.run.smithi076.stdout:(8/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 29 MB/s | 17 MB 00:00 2024-09-16T03:36:54.456 INFO:teuthology.orchestra.run.smithi076.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 648 kB/s | 135 kB 00:00 2024-09-16T03:36:54.463 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:36:54.463 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:36:54.487 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:41 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:36:54.514 INFO:teuthology.orchestra.run.smithi076.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 3.4 MB/s | 1.2 MB 00:00 2024-09-16T03:36:54.540 INFO:teuthology.orchestra.run.smithi076.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-16T03:36:54.582 INFO:teuthology.orchestra.run.smithi076.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 829 kB/s | 125 kB 00:00 2024-09-16T03:36:54.632 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:36:54.635 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:36:54.635 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-16T03:36:54.635 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:36:54.635 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:36:54.635 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T03:36:54.635 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-16T03:36:54.635 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T03:36:54.636 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T03:36:54.636 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T03:36:54.636 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T03:36:54.636 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T03:36:54.636 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T03:36:54.636 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T03:36:54.636 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T03:36:54.636 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T03:36:54.636 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T03:36:54.637 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T03:36:54.638 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T03:36:54.638 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T03:36:54.638 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T03:36:54.638 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T03:36:54.638 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T03:36:54.638 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T03:36:54.638 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T03:36:54.638 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T03:36:54.638 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T03:36:54.638 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T03:36:54.639 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T03:36:54.639 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T03:36:54.639 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T03:36:54.639 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T03:36:54.639 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:54.640 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:36:54.640 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:36:54.640 INFO:teuthology.orchestra.run.smithi121.stdout:Install 37 Packages 2024-09-16T03:36:54.640 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:36:54.641 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 30 M 2024-09-16T03:36:54.641 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 108 M 2024-09-16T03:36:54.641 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:36:54.673 INFO:teuthology.orchestra.run.smithi076.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-16T03:36:54.690 INFO:teuthology.orchestra.run.smithi076.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-16T03:36:54.716 INFO:teuthology.orchestra.run.smithi076.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 980 kB/s | 172 kB 00:00 2024-09-16T03:36:54.741 INFO:teuthology.orchestra.run.smithi076.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.1 MB/s | 77 kB 00:00 2024-09-16T03:36:54.758 INFO:teuthology.orchestra.run.smithi076.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 757 kB/s | 51 kB 00:00 2024-09-16T03:36:54.775 INFO:teuthology.orchestra.run.smithi076.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 709 kB/s | 42 kB 00:00 2024-09-16T03:36:54.817 INFO:teuthology.orchestra.run.smithi076.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-16T03:36:54.842 INFO:teuthology.orchestra.run.smithi076.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 649 kB/s | 43 kB 00:00 2024-09-16T03:36:54.867 INFO:teuthology.orchestra.run.smithi076.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-16T03:36:54.884 INFO:teuthology.orchestra.run.smithi076.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 651 kB/s | 11 kB 00:00 2024-09-16T03:36:54.901 INFO:teuthology.orchestra.run.smithi076.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-16T03:36:54.943 INFO:teuthology.orchestra.run.smithi076.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 554 kB/s | 23 kB 00:00 2024-09-16T03:36:54.945 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:36:54.945 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:36:54.968 INFO:teuthology.orchestra.run.smithi076.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-16T03:36:54.986 INFO:teuthology.orchestra.run.smithi076.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 467 kB/s | 19 kB 00:00 2024-09-16T03:36:55.002 INFO:teuthology.orchestra.run.smithi076.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 582 kB/s | 20 kB 00:00 2024-09-16T03:36:55.027 INFO:teuthology.orchestra.run.smithi076.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-09-16T03:36:55.056 INFO:teuthology.orchestra.run.smithi076.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.4 MB/s | 79 kB 00:00 2024-09-16T03:36:55.082 INFO:teuthology.orchestra.run.smithi076.stdout:(30/37): python3-pecan-1.4.2-3.el9.noarch.rpm 5.7 MB/s | 272 kB 00:00 2024-09-16T03:36:55.099 INFO:teuthology.orchestra.run.smithi076.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2024-09-16T03:36:55.124 INFO:teuthology.orchestra.run.smithi076.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-16T03:36:55.141 INFO:teuthology.orchestra.run.smithi076.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 855 kB/s | 36 kB 00:00 2024-09-16T03:36:55.166 INFO:teuthology.orchestra.run.smithi076.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-09-16T03:36:55.192 INFO:teuthology.orchestra.run.smithi076.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-16T03:36:55.225 INFO:teuthology.orchestra.run.smithi076.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-16T03:36:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:36:55.306 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:36:55.367 INFO:teuthology.orchestra.run.smithi076.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 36 kB/s | 22 kB 00:00 2024-09-16T03:36:55.371 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:55.371 INFO:teuthology.orchestra.run.smithi076.stdout:Total 14 MB/s | 30 MB 00:02 2024-09-16T03:36:55.465 INFO:teuthology.orchestra.run.smithi121.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 48 kB/s | 6.5 kB 00:00 2024-09-16T03:36:55.517 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:36:55.559 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:36:55.559 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:36:55.624 INFO:teuthology.orchestra.run.smithi121.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 5.1 MB/s | 1.5 MB 00:00 2024-09-16T03:36:55.699 INFO:teuthology.orchestra.run.smithi121.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.4 MB/s | 2.4 MB 00:00 2024-09-16T03:36:55.829 INFO:teuthology.orchestra.run.smithi121.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 1.9 MB/s | 249 kB 00:00 2024-09-16T03:36:55.900 INFO:teuthology.orchestra.run.smithi121.stdout:(5/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-16T03:36:56.009 INFO:teuthology.orchestra.run.smithi121.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-16T03:36:56.042 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:36:56.042 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:36:56.051 INFO:teuthology.orchestra.run.smithi121.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T03:36:56.217 INFO:teuthology.orchestra.run.smithi121.stdout:(8/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 28 MB/s | 17 MB 00:00 2024-09-16T03:36:56.243 INFO:teuthology.orchestra.run.smithi121.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 705 kB/s | 135 kB 00:00 2024-09-16T03:36:56.276 INFO:teuthology.orchestra.run.smithi121.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 3.3 MB/s | 1.2 MB 00:00 2024-09-16T03:36:56.302 INFO:teuthology.orchestra.run.smithi121.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T03:36:56.305 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:36:56.344 INFO:teuthology.orchestra.run.smithi121.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 995 kB/s | 125 kB 00:00 2024-09-16T03:36:56.383 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T03:36:56.427 INFO:teuthology.orchestra.run.smithi121.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-16T03:36:56.432 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T03:36:56.452 INFO:teuthology.orchestra.run.smithi121.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-16T03:36:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T03:36:56.478 INFO:teuthology.orchestra.run.smithi121.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-16T03:36:56.503 INFO:teuthology.orchestra.run.smithi121.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 856 kB/s | 172 kB 00:00 2024-09-16T03:36:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-09-16T03:36:56.550 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T03:36:56.553 INFO:teuthology.orchestra.run.smithi121.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-09-16T03:36:56.609 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T03:36:56.612 INFO:teuthology.orchestra.run.smithi121.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 748 kB/s | 43 kB 00:00 2024-09-16T03:36:56.637 INFO:teuthology.orchestra.run.smithi121.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-16T03:36:56.853 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T03:36:56.853 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T03:36:56.853 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T03:36:56.854 INFO:teuthology.orchestra.run.smithi121.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 140 kB/s | 22 kB 00:00 2024-09-16T03:36:56.854 INFO:teuthology.orchestra.run.smithi121.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-16T03:36:56.854 INFO:teuthology.orchestra.run.smithi121.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 767 kB/s | 18 kB 00:00 2024-09-16T03:36:56.854 INFO:teuthology.orchestra.run.smithi121.stdout:(24/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 127 kB/s | 11 kB 00:00 2024-09-16T03:36:56.854 INFO:teuthology.orchestra.run.smithi121.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-09-16T03:36:56.855 INFO:teuthology.orchestra.run.smithi121.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-16T03:36:56.855 INFO:teuthology.orchestra.run.smithi121.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-16T03:36:56.855 INFO:teuthology.orchestra.run.smithi121.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-16T03:36:56.872 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T03:36:56.873 INFO:teuthology.orchestra.run.smithi121.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-16T03:36:56.890 INFO:teuthology.orchestra.run.smithi121.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-16T03:36:56.907 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T03:36:56.907 INFO:teuthology.orchestra.run.smithi121.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 491 kB/s | 16 kB 00:00 2024-09-16T03:36:56.924 INFO:teuthology.orchestra.run.smithi121.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-16T03:36:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T03:36:56.949 INFO:teuthology.orchestra.run.smithi121.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-16T03:36:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T03:36:56.975 INFO:teuthology.orchestra.run.smithi121.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-16T03:36:57.004 INFO:teuthology.orchestra.run.smithi121.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.9 MB/s | 230 kB 00:00 2024-09-16T03:36:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T03:36:57.029 INFO:teuthology.orchestra.run.smithi121.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 373 kB/s | 20 kB 00:00 2024-09-16T03:36:57.045 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T03:36:57.054 INFO:teuthology.orchestra.run.smithi121.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.0 MB/s | 427 kB 00:00 2024-09-16T03:36:57.057 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:36:57.057 INFO:teuthology.orchestra.run.smithi121.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-16T03:36:57.098 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T03:36:57.131 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T03:36:57.180 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T03:36:57.213 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:36:57.221 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T03:36:57.252 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:36:57.252 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:36:57.301 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T03:36:57.372 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:36:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T03:36:57.445 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T03:36:57.482 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T03:36:57.514 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T03:36:57.536 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T03:36:57.584 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T03:36:57.589 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T03:36:57.628 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T03:36:57.638 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T03:36:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T03:36:57.676 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T03:36:57.730 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:36:57.731 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:36:57.740 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T03:36:57.761 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:36:57.762 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T03:36:57.762 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T03:36:57.762 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T03:36:57.762 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T03:36:57.840 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T03:36:57.881 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T03:36:57.916 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T03:36:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T03:36:57.946 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T03:36:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T03:36:57.981 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T03:36:58.014 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T03:36:58.042 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T03:36:58.055 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T03:36:58.089 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T03:36:58.099 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T03:36:58.099 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:58.099 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:36:58.100 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:58.100 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:36:58.100 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T03:36:58.100 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T03:36:58.100 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:58.100 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:36:58.124 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T03:36:58.139 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T03:36:58.172 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T03:36:58.181 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T03:36:58.205 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T03:36:58.277 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T03:36:58.296 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T03:36:58.408 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph 2024-09-16T03:36:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T03:36:58.464 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T03:36:58.492 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:36:58.495 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T03:36:58.518 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:36:58.518 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:36:58.518 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T03:36:58.518 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:36:58.518 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:36:58.518 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T03:36:58.587 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T03:36:58.640 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T03:36:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T03:36:58.880 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T03:36:58.929 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T03:36:58.998 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T03:36:59.003 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:46 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:36:59.072 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:36:59.086 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T03:36:59.128 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T03:36:59.140 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T03:36:59.155 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:36:59.157 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:36:59.157 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T03:36:59.157 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T03:36:59.158 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T03:36:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T03:36:59.160 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T03:36:59.161 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T03:36:59.161 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:59.161 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:36:59.161 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:36:59.161 INFO:teuthology.orchestra.run.smithi103.stdout:Install 37 Packages 2024-09-16T03:36:59.161 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:36:59.161 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 30 M 2024-09-16T03:36:59.162 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 108 M 2024-09-16T03:36:59.162 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:36:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T03:36:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T03:36:59.243 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T03:36:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T03:36:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T03:36:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T03:36:59.424 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T03:36:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:36:59.464 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:36:59.464 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:36:59.464 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T03:36:59.465 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:36:59.465 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:36:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:36:59.517 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T03:36:59.609 INFO:teuthology.orchestra.run.smithi103.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 48 kB/s | 6.5 kB 00:00 2024-09-16T03:36:59.619 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T03:36:59.660 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T03:36:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T03:36:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T03:36:59.767 INFO:teuthology.orchestra.run.smithi103.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 5.2 MB/s | 1.5 MB 00:00 2024-09-16T03:36:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T03:36:59.810 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T03:36:59.847 INFO:teuthology.orchestra.run.smithi103.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.4 MB/s | 2.4 MB 00:00 2024-09-16T03:36:59.850 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T03:36:59.885 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T03:36:59.905 INFO:teuthology.orchestra.run.smithi103.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 4.2 MB/s | 249 kB 00:00 2024-09-16T03:36:59.920 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T03:36:59.959 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:36:59.960 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T03:36:59.987 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:36:59.987 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:36:59.987 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T03:36:59.987 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:36:59.987 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:36:59.987 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:36:59.993 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T03:37:00.065 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T03:37:00.106 INFO:teuthology.orchestra.run.smithi103.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-16T03:37:00.172 INFO:teuthology.orchestra.run.smithi103.stdout:(6/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 8.4 MB/s | 4.7 MB 00:00 2024-09-16T03:37:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T03:37:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T03:37:00.344 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T03:37:00.365 INFO:teuthology.orchestra.run.smithi103.stdout:(7/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 28 MB/s | 17 MB 00:00 2024-09-16T03:37:00.383 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T03:37:00.415 INFO:teuthology.orchestra.run.smithi103.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 4.0 MB/s | 1.2 MB 00:00 2024-09-16T03:37:00.426 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:37:00.440 INFO:teuthology.orchestra.run.smithi103.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 399 kB/s | 106 kB 00:00 2024-09-16T03:37:00.440 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T03:37:00.450 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:37:00.450 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:00.450 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T03:37:00.450 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:37:00.451 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:37:00.451 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:00.465 INFO:teuthology.orchestra.run.smithi103.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-16T03:37:00.496 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T03:37:00.549 INFO:teuthology.orchestra.run.smithi103.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-09-16T03:37:00.575 INFO:teuthology.orchestra.run.smithi103.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 645 kB/s | 135 kB 00:00 2024-09-16T03:37:00.600 INFO:teuthology.orchestra.run.smithi103.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-09-16T03:37:00.642 INFO:teuthology.orchestra.run.smithi103.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-16T03:37:00.659 INFO:teuthology.orchestra.run.smithi103.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 413 kB/s | 35 kB 00:00 2024-09-16T03:37:00.675 INFO:teuthology.orchestra.run.smithi103.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.5 MB/s | 51 kB 00:00 2024-09-16T03:37:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T03:37:00.685 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:37:00.701 INFO:teuthology.orchestra.run.smithi103.stdout:(17/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-16T03:37:00.709 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:37:00.709 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:00.709 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T03:37:00.709 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:37:00.709 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:37:00.709 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:00.727 INFO:teuthology.orchestra.run.smithi103.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 615 kB/s | 42 kB 00:00 2024-09-16T03:37:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T03:37:00.743 INFO:teuthology.orchestra.run.smithi103.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 329 kB/s | 22 kB 00:00 2024-09-16T03:37:00.774 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:37:00.785 INFO:teuthology.orchestra.run.smithi103.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 744 kB/s | 43 kB 00:00 2024-09-16T03:37:00.810 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T03:37:00.810 INFO:teuthology.orchestra.run.smithi103.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-09-16T03:37:00.828 INFO:teuthology.orchestra.run.smithi103.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 638 kB/s | 11 kB 00:00 2024-09-16T03:37:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-09-16T03:37:00.870 INFO:teuthology.orchestra.run.smithi103.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 422 kB/s | 18 kB 00:00 2024-09-16T03:37:00.887 INFO:teuthology.orchestra.run.smithi103.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 687 kB/s | 23 kB 00:00 2024-09-16T03:37:00.904 INFO:teuthology.orchestra.run.smithi103.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 581 kB/s | 19 kB 00:00 2024-09-16T03:37:00.905 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:37:00.906 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T03:37:00.921 INFO:teuthology.orchestra.run.smithi103.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 583 kB/s | 20 kB 00:00 2024-09-16T03:37:00.933 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:37:00.933 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:00.933 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T03:37:00.933 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:37:00.933 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:37:00.933 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:00.938 INFO:teuthology.orchestra.run.smithi103.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-16T03:37:00.948 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T03:37:00.963 INFO:teuthology.orchestra.run.smithi103.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-09-16T03:37:00.988 INFO:teuthology.orchestra.run.smithi103.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-16T03:37:01.006 INFO:teuthology.orchestra.run.smithi103.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 390 kB/s | 16 kB 00:00 2024-09-16T03:37:01.031 INFO:teuthology.orchestra.run.smithi103.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-16T03:37:01.048 INFO:teuthology.orchestra.run.smithi103.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 853 kB/s | 36 kB 00:00 2024-09-16T03:37:01.063 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T03:37:01.073 INFO:teuthology.orchestra.run.smithi103.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-16T03:37:01.098 INFO:teuthology.orchestra.run.smithi103.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-09-16T03:37:01.123 INFO:teuthology.orchestra.run.smithi103.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-09-16T03:37:01.157 INFO:teuthology.orchestra.run.smithi103.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-16T03:37:01.161 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:01.161 INFO:teuthology.orchestra.run.smithi103.stdout:Total 15 MB/s | 30 MB 00:02 2024-09-16T03:37:01.220 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T03:37:01.249 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:37:01.276 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:37:01.276 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:01.276 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T03:37:01.276 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:37:01.276 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:37:01.276 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:01.306 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:01.346 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:01.346 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:01.365 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:37:01.388 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:37:01.388 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:01.388 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T03:37:01.388 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:37:01.388 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:37:01.389 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:37:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:37:01.647 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:01.647 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T03:37:01.648 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:37:01.648 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:37:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:01.712 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:37:01.821 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:01.821 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T03:37:02.709 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:37:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:37:02.736 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:02.737 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T03:37:02.737 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:37:02.737 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:37:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:03.165 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:37:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:37:03.200 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:03.201 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T03:37:03.201 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:37:03.201 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:37:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T03:37:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T03:37:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T03:37:03.370 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T03:37:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T03:37:03.427 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:37:03.451 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:37:03.451 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:03.451 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T03:37:03.451 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:37:03.451 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:37:03.451 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T03:37:03.515 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:37:03.521 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T03:37:03.522 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:03.522 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:03.522 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:03.522 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:03.522 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:03.522 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:03.522 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:03.522 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:03.522 INFO:teuthology.orchestra.run.smithi119.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T03:37:03.522 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T03:37:03.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T03:37:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T03:37:03.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T03:37:03.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T03:37:03.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T03:37:03.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T03:37:03.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T03:37:03.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T03:37:03.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T03:37:03.525 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:03.525 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:03.564 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:37:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T03:37:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T03:37:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T03:37:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T03:37:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T03:37:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T03:37:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T03:37:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T03:37:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T03:37:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T03:37:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T03:37:03.567 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T03:37:03.567 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T03:37:03.567 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T03:37:03.567 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T03:37:03.567 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T03:37:03.567 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T03:37:03.568 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T03:37:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T03:37:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T03:37:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T03:37:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T03:37:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T03:37:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T03:37:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T03:37:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T03:37:03.639 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T03:37:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T03:37:03.733 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T03:37:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T03:37:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T03:37:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T03:37:03.877 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-base 2024-09-16T03:37:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T03:37:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T03:37:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T03:37:03.996 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T03:37:04.029 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T03:37:04.102 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T03:37:04.241 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T03:37:04.322 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T03:37:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T03:37:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T03:37:04.442 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:04.485 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T03:37:04.533 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:04.541 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T03:37:04.542 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T03:37:04.543 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T03:37:04.544 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T03:37:04.545 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T03:37:04.545 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T03:37:04.545 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:04.545 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:04.545 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T03:37:04.577 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:04.577 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-16T03:37:04.577 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:04.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install cephadm 2024-09-16T03:37:04.728 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T03:37:04.775 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T03:37:04.854 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T03:37:04.929 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-base 2024-09-16T03:37:04.942 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T03:37:04.986 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T03:37:05.114 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T03:37:05.205 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:05.287 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T03:37:05.318 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:37:05.335 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:05.336 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:05.336 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-16T03:37:05.336 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:05.336 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:05.336 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T03:37:05.336 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:05.337 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:05.337 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:05.337 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-16T03:37:05.337 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:05.337 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 765 k 2024-09-16T03:37:05.337 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 771 k 2024-09-16T03:37:05.337 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:37:05.344 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:05.344 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T03:37:05.345 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:37:05.345 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:37:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:05.428 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:37:05.428 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T03:37:05.428 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T03:37:05.428 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T03:37:05.428 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T03:37:05.428 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T03:37:05.428 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T03:37:05.428 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T03:37:05.428 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T03:37:05.428 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T03:37:05.429 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T03:37:05.430 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T03:37:05.431 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T03:37:05.431 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T03:37:05.494 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:05.584 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:05.626 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-16T03:37:05.626 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:05.627 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:05.627 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-16T03:37:05.627 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:05.627 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T03:37:05.628 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:05.632 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:05.632 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:05.638 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:05.639 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:05.708 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install cephadm 2024-09-16T03:37:05.709 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:06.187 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:06.271 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:06.272 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:06.403 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:06.403 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 765 k 2024-09-16T03:37:06.404 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 771 k 2024-09-16T03:37:06.405 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:37:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T03:37:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:06.461 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T03:37:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T03:37:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:06.464 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:06.689 INFO:teuthology.orchestra.run.smithi076.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-16T03:37:06.690 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:06.690 INFO:teuthology.orchestra.run.smithi076.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-16T03:37:06.690 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:37:06.695 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:06.695 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:37:06.695 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:37:06.702 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:37:06.702 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:37:06.778 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:37:06.794 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:37:06.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-09-16T03:37:06.821 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:37:06.821 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:06.821 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T03:37:06.822 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:37:06.822 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:37:06.822 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:06.935 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:06.936 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:06.936 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:06.936 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:06.936 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:06.936 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:07.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T03:37:07.176 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:07.262 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:37:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:07.286 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:37:07.286 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:07.286 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T03:37:07.286 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:37:07.286 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:37:07.286 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:07.379 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:07.470 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:07.511 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:37:07.513 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-16T03:37:07.513 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:37:07.536 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:07.536 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T03:37:07.536 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:37:07.536 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:37:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:07.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-09-16T03:37:07.601 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:37:07.649 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:07.757 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:07.892 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:07.893 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 145 k 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 435 k 2024-09-16T03:37:07.894 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:07.900 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:37:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:07.900 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 738 kB/s | 145 kB 00:00 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi119.stdout:Total 731 kB/s | 145 kB 00:00 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:08.118 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T03:37:08.135 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:08.155 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:08.156 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:08.265 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:08.265 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:08.265 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 765 k 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 771 k 2024-09-16T03:37:08.266 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:08.318 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:08.408 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:08.433 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:08.433 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:08.433 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T03:37:08.433 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:08.526 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-16T03:37:08.527 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:08.527 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-09-16T03:37:08.527 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:08.532 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:08.532 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:08.538 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:08.539 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:08.688 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:08.822 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:08.822 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:08.822 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-09-16T03:37:08.822 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:08.823 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:37:08.823 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T03:37:08.823 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:08.823 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:37:08.823 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:08.823 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T03:37:08.823 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:08.824 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 145 k 2024-09-16T03:37:08.824 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 435 k 2024-09-16T03:37:08.824 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:37:09.029 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 708 kB/s | 145 kB 00:00 2024-09-16T03:37:09.029 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:09.030 INFO:teuthology.orchestra.run.smithi076.stdout:Total 703 kB/s | 145 kB 00:00 2024-09-16T03:37:09.030 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:37:09.037 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:37:09.037 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:37:09.054 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:09.094 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:37:09.094 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:37:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:37:09.338 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:09.364 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:09.364 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:09.364 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T03:37:09.364 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:09.401 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:09.402 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:09.402 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:09.402 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:09.402 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:09.402 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:09.465 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:09.524 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:37:09.524 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T03:37:09.524 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T03:37:09.524 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T03:37:09.524 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T03:37:09.524 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T03:37:09.524 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T03:37:09.524 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T03:37:09.524 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T03:37:09.524 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T03:37:09.525 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T03:37:09.526 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T03:37:09.527 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T03:37:09.527 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T03:37:09.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr 2024-09-16T03:37:09.719 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:09.719 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:09.719 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:09.719 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:09.719 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:09.719 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:09.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T03:37:10.203 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:10.294 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:10.295 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:37:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:10.295 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:10.336 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:10.337 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-16T03:37:10.337 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:10.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T03:37:10.507 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:10.533 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T03:37:10.533 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:10.533 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:10.533 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T03:37:10.534 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T03:37:10.535 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:10.536 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:10.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr 2024-09-16T03:37:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-16T03:37:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T03:37:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-16T03:37:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:10.646 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 145 k 2024-09-16T03:37:10.646 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 435 k 2024-09-16T03:37:10.646 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:10.948 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base 2024-09-16T03:37:10.965 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:10.993 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 418 kB/s | 145 kB 00:00 2024-09-16T03:37:10.995 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:10.995 INFO:teuthology.orchestra.run.smithi121.stdout:Total 416 kB/s | 145 kB 00:00 2024-09-16T03:37:10.995 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:11.002 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:11.002 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:11.060 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:11.098 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:11.099 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:11.099 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-16T03:37:11.099 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:11.099 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:11.100 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T03:37:11.100 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-16T03:37:11.100 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T03:37:11.100 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T03:37:11.100 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T03:37:11.100 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T03:37:11.100 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T03:37:11.100 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T03:37:11.100 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T03:37:11.100 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T03:37:11.101 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T03:37:11.101 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T03:37:11.101 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T03:37:11.101 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:11.101 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:11.101 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:11.101 INFO:teuthology.orchestra.run.smithi119.stdout:Install 12 Packages 2024-09-16T03:37:11.101 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:11.102 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 8.7 M 2024-09-16T03:37:11.102 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 104 M 2024-09-16T03:37:11.102 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:11.132 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:11.203 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:11.225 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:11.252 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:11.268 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:11.270 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T03:37:11.270 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:11.277 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:11.277 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:11.277 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T03:37:11.277 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:11.355 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T03:37:11.519 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:11.609 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:11.651 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:11.652 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T03:37:11.652 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:11.710 INFO:teuthology.orchestra.run.smithi119.stdout:(1/12): ceph-prometheus-alerts-19.1.1-326.g8921 99 kB/s | 16 kB 00:00 2024-09-16T03:37:11.717 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install cephadm 2024-09-16T03:37:11.736 INFO:teuthology.orchestra.run.smithi119.stdout:(2/12): ceph-grafana-dashboards-19.1.1-326.g892 166 kB/s | 32 kB 00:00 2024-09-16T03:37:11.912 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:11.937 INFO:teuthology.orchestra.run.smithi119.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-16T03:37:12.003 INFO:teuthology.orchestra.run.smithi119.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-09-16T03:37:12.044 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:12.045 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:12.045 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T03:37:12.045 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:12.045 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:37:12.045 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T03:37:12.045 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T03:37:12.045 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T03:37:12.045 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T03:37:12.045 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout:Install 12 Packages 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:12.047 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 8.7 M 2024-09-16T03:37:12.047 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 104 M 2024-09-16T03:37:12.047 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:37:12.062 INFO:teuthology.orchestra.run.smithi119.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-16T03:37:12.078 INFO:teuthology.orchestra.run.smithi119.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-16T03:37:12.112 INFO:teuthology.orchestra.run.smithi119.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-09-16T03:37:12.171 INFO:teuthology.orchestra.run.smithi119.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-16T03:37:12.188 INFO:teuthology.orchestra.run.smithi119.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-16T03:37:12.230 INFO:teuthology.orchestra.run.smithi119.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-16T03:37:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:12.245 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:12.245 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:12.275 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:12.305 INFO:teuthology.orchestra.run.smithi119.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.0 MB/s | 2.0 MB 00:00 2024-09-16T03:37:12.388 INFO:teuthology.orchestra.run.smithi119.stdout:(12/12): ceph-mgr-dashboard-19.1.1-326.g8921c8e 4.2 MB/s | 3.6 MB 00:00 2024-09-16T03:37:12.392 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:12.393 INFO:teuthology.orchestra.run.smithi119.stdout:Total 6.7 MB/s | 8.7 MB 00:01 2024-09-16T03:37:12.407 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:12.407 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T03:37:12.407 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:12.407 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:37:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T03:37:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:12.407 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:37:12.408 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:12.408 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T03:37:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:12.408 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 765 k 2024-09-16T03:37:12.408 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 771 k 2024-09-16T03:37:12.408 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:37:12.416 INFO:teuthology.orchestra.run.smithi076.stdout:(1/12): ceph-grafana-dashboards-19.1.1-326.g892 217 kB/s | 32 kB 00:00 2024-09-16T03:37:12.441 INFO:teuthology.orchestra.run.smithi076.stdout:(2/12): ceph-prometheus-alerts-19.1.1-326.g8921 95 kB/s | 16 kB 00:00 2024-09-16T03:37:12.443 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:12.460 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:12.461 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:12.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-09-16T03:37:12.567 INFO:teuthology.orchestra.run.smithi076.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 269 kB 00:00 2024-09-16T03:37:12.617 INFO:teuthology.orchestra.run.smithi076.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-09-16T03:37:12.692 INFO:teuthology.orchestra.run.smithi076.stdout:(5/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 8.4 MB/s | 3.6 MB 00:00 2024-09-16T03:37:12.697 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:12.698 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:12.715 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.4 MB/s | 765 kB 00:00 2024-09-16T03:37:12.715 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:12.716 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.4 MB/s | 765 kB 00:00 2024-09-16T03:37:12.716 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:12.721 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:12.721 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:12.727 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:12.727 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:12.734 INFO:teuthology.orchestra.run.smithi076.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.6 MB/s | 548 kB 00:00 2024-09-16T03:37:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-09-16T03:37:12.793 INFO:teuthology.orchestra.run.smithi076.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-16T03:37:12.800 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:12.818 INFO:teuthology.orchestra.run.smithi076.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T03:37:12.844 INFO:teuthology.orchestra.run.smithi076.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 3.0 MB/s | 863 kB 00:00 2024-09-16T03:37:12.877 INFO:teuthology.orchestra.run.smithi076.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-09-16T03:37:12.936 INFO:teuthology.orchestra.run.smithi076.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 10 MB/s | 2.0 MB 00:00 2024-09-16T03:37:12.939 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:12.939 INFO:teuthology.orchestra.run.smithi076.stdout:Total 9.7 MB/s | 8.7 MB 00:00 2024-09-16T03:37:12.986 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:37:13.002 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:37:13.003 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:37:13.057 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:13.190 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:13.191 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-16T03:37:13.191 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:13.211 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:37:13.212 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:37:13.233 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:13.242 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:13.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T03:37:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T03:37:13.334 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:13.367 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T03:37:13.412 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T03:37:13.451 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T03:37:13.557 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T03:37:13.694 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T03:37:13.722 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:37:13.789 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T03:37:13.812 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:13.847 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T03:37:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T03:37:13.876 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:37:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T03:37:13.907 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T03:37:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T03:37:13.943 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T03:37:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T03:37:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T03:37:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T03:37:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T03:37:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:13.946 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:13.946 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:13.946 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-09-16T03:37:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:13.947 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 8.7 M 2024-09-16T03:37:13.947 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 104 M 2024-09-16T03:37:13.947 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T03:37:13.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:13.983 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:13.983 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:13.983 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:13.984 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:13.984 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:14.057 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T03:37:14.182 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T03:37:14.201 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T03:37:14.276 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T03:37:14.313 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): ceph-grafana-dashboards-19.1.1-326.g892 205 kB/s | 32 kB 00:00 2024-09-16T03:37:14.320 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T03:37:14.339 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): ceph-prometheus-alerts-19.1.1-326.g8921 91 kB/s | 16 kB 00:00 2024-09-16T03:37:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:37:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T03:37:14.556 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 9.0 MB/s | 3.6 MB 00:00 2024-09-16T03:37:14.589 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-09-16T03:37:14.631 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-09-16T03:37:14.682 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.8 MB/s | 548 kB 00:00 2024-09-16T03:37:14.699 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 298 kB/s | 19 kB 00:00 2024-09-16T03:37:14.732 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.3 MB/s | 144 kB 00:00 2024-09-16T03:37:14.757 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T03:37:14.765 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:01 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:14.799 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-16T03:37:14.858 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-09-16T03:37:14.883 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.6 MB/s | 863 kB 00:00 2024-09-16T03:37:14.886 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:14.887 INFO:teuthology.orchestra.run.smithi121.stdout:Total 9.2 MB/s | 8.7 MB 00:00 2024-09-16T03:37:14.899 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:14.899 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:14.900 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-16T03:37:14.900 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:14.900 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:37:14.900 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T03:37:14.900 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:14.900 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:37:14.900 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:14.900 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T03:37:14.900 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:14.900 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 145 k 2024-09-16T03:37:14.901 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 435 k 2024-09-16T03:37:14.901 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:37:14.934 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:14.951 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:14.951 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:14.954 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T03:37:14.992 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:37:15.115 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 677 kB/s | 145 kB 00:00 2024-09-16T03:37:15.115 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:15.115 INFO:teuthology.orchestra.run.smithi103.stdout:Total 673 kB/s | 145 kB 00:00 2024-09-16T03:37:15.116 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:15.123 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:15.123 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:15.165 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:15.165 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:15.180 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:15.181 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:15.334 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:15.416 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:15.437 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T03:37:15.440 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:15.440 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:15.441 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T03:37:15.441 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:15.473 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:37:15.696 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T03:37:15.857 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T03:37:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T03:37:15.982 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T03:37:16.113 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T03:37:16.207 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T03:37:16.279 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T03:37:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:37:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T03:37:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T03:37:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T03:37:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T03:37:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T03:37:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T03:37:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T03:37:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T03:37:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:37:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T03:37:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T03:37:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:37:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:16.393 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:16.393 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:16.654 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr 2024-09-16T03:37:16.691 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T03:37:16.691 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:16.691 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T03:37:16.692 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T03:37:16.693 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:16.693 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:16.864 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:37:16.864 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T03:37:16.864 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T03:37:16.864 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T03:37:16.865 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T03:37:16.865 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T03:37:16.865 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T03:37:16.865 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T03:37:16.865 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T03:37:16.865 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:37:16.865 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T03:37:16.865 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T03:37:16.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T03:37:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T03:37:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:17.206 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T03:37:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:17.208 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:17.238 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:04 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T03:37:17.374 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:17.375 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T03:37:17.375 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:37:17.438 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T03:37:17.447 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T03:37:17.504 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:17.644 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:17.645 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-09-16T03:37:17.645 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T03:37:17.646 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T03:37:17.647 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T03:37:17.647 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T03:37:17.647 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:17.647 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:17.647 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-09-16T03:37:17.647 INFO:teuthology.orchestra.run.smithi119.stdout:Install 12 Packages 2024-09-16T03:37:17.647 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:17.647 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 38 M 2024-09-16T03:37:17.648 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 212 M 2024-09-16T03:37:17.648 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:17.913 INFO:teuthology.orchestra.run.smithi119.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-16T03:37:17.997 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:04 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:18.010 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:05 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:18.080 INFO:teuthology.orchestra.run.smithi119.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-16T03:37:18.129 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:18.130 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:18.130 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T03:37:18.130 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T03:37:18.131 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T03:37:18.132 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T03:37:18.132 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:18.132 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:37:18.132 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:18.132 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-09-16T03:37:18.132 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:18.132 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 8.7 M 2024-09-16T03:37:18.133 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 104 M 2024-09-16T03:37:18.133 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:37:18.144 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:18.145 INFO:teuthology.orchestra.run.smithi076.stdout:====================================================================================== 2024-09-16T03:37:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T03:37:18.145 INFO:teuthology.orchestra.run.smithi076.stdout:====================================================================================== 2024-09-16T03:37:18.145 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:37:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T03:37:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:18.147 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:37:18.147 INFO:teuthology.orchestra.run.smithi076.stdout:====================================================================================== 2024-09-16T03:37:18.147 INFO:teuthology.orchestra.run.smithi076.stdout:Install 12 Packages 2024-09-16T03:37:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:18.147 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 38 M 2024-09-16T03:37:18.148 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 212 M 2024-09-16T03:37:18.148 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:37:18.211 INFO:teuthology.orchestra.run.smithi119.stdout:(3/12): ceph-mgr-diskprediction-local-19.1.1-32 16 MB/s | 7.4 MB 00:00 2024-09-16T03:37:18.511 INFO:teuthology.orchestra.run.smithi076.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 990 kB/s | 191 kB 00:00 2024-09-16T03:37:18.544 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): ceph-prometheus-alerts-19.1.1-326.g8921 102 kB/s | 16 kB 00:00 2024-09-16T03:37:18.561 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): ceph-grafana-dashboards-19.1.1-326.g892 179 kB/s | 32 kB 00:00 2024-09-16T03:37:18.573 INFO:teuthology.orchestra.run.smithi119.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 50 kB/s | 33 kB 00:00 2024-09-16T03:37:18.578 INFO:teuthology.orchestra.run.smithi076.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.0 MB/s | 801 kB 00:00 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T03:37:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T03:37:18.703 INFO:teuthology.orchestra.run.smithi119.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-16T03:37:18.736 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 269 kB 00:00 2024-09-16T03:37:18.778 INFO:teuthology.orchestra.run.smithi076.stdout:(3/12): ceph-mgr-diskprediction-local-19.1.1-32 16 MB/s | 7.4 MB 00:00 2024-09-16T03:37:18.795 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-09-16T03:37:18.803 INFO:teuthology.orchestra.run.smithi076.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 112 kB/s | 33 kB 00:00 2024-09-16T03:37:18.878 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 6.4 MB/s | 548 kB 00:00 2024-09-16T03:37:18.937 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 6.4 MB/s | 3.6 MB 00:00 2024-09-16T03:37:18.962 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 232 kB/s | 19 kB 00:00 2024-09-16T03:37:18.996 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.3 MB/s | 144 kB 00:00 2024-09-16T03:37:19.004 INFO:teuthology.orchestra.run.smithi076.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-16T03:37:19.021 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-09-16T03:37:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T03:37:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:19.028 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T03:37:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T03:37:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T03:37:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T03:37:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T03:37:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T03:37:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T03:37:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T03:37:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T03:37:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:19.029 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:19.062 INFO:teuthology.orchestra.run.smithi076.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-16T03:37:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-16T03:37:19.132 INFO:teuthology.orchestra.run.smithi119.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 69 kB/s | 38 kB 00:00 2024-09-16T03:37:19.197 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.9 MB/s | 2.0 MB 00:00 2024-09-16T03:37:19.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T03:37:19.373 INFO:teuthology.orchestra.run.smithi119.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.3 MB/s | 3.0 MB 00:01 2024-09-16T03:37:19.537 INFO:teuthology.orchestra.run.smithi076.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 515 kB/s | 245 kB 00:00 2024-09-16T03:37:19.663 INFO:teuthology.orchestra.run.smithi076.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.8 MB/s | 3.0 MB 00:01 2024-09-16T03:37:19.796 INFO:teuthology.orchestra.run.smithi076.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 250 kB 00:00 2024-09-16T03:37:19.818 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:19.951 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:19.952 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-09-16T03:37:19.952 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-16T03:37:19.952 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-09-16T03:37:19.952 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:19.952 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T03:37:19.953 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:19.954 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:19.954 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-09-16T03:37:19.954 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-09-16T03:37:19.954 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:19.954 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 38 M 2024-09-16T03:37:19.954 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 212 M 2024-09-16T03:37:19.954 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:20.013 INFO:teuthology.orchestra.run.smithi076.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.9 MB/s | 4.9 MB 00:01 2024-09-16T03:37:20.015 INFO:teuthology.orchestra.run.smithi119.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 277 kB/s | 245 kB 00:00 2024-09-16T03:37:20.166 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-16T03:37:20.225 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.9 MB/s | 801 kB 00:00 2024-09-16T03:37:20.250 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-09-16T03:37:20.292 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-09-16T03:37:20.334 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 921 kB/s | 38 kB 00:00 2024-09-16T03:37:20.372 INFO:teuthology.orchestra.run.smithi076.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.1 MB/s | 5.1 MB 00:00 2024-09-16T03:37:20.382 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 525 kB/s | 863 kB 00:01 2024-09-16T03:37:20.387 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:20.387 INFO:teuthology.orchestra.run.smithi103.stdout:Total 3.8 MB/s | 8.7 MB 00:02 2024-09-16T03:37:20.439 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:20.458 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:20.458 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:20.476 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-09-16T03:37:20.593 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): ceph-mgr-diskprediction-local-19.1.1-32 13 MB/s | 7.4 MB 00:00 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-09-16T03:37:20.660 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-16T03:37:20.665 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:20.666 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:20.682 INFO:teuthology.orchestra.run.smithi119.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 2.5 MB/s | 4.9 MB 00:01 2024-09-16T03:37:20.727 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 13 MB/s | 4.9 MB 00:00 2024-09-16T03:37:20.791 INFO:teuthology.orchestra.run.smithi119.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.6 MB/s | 5.1 MB 00:01 2024-09-16T03:37:20.977 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi121.stdout:Total 35 MB/s | 38 MB 00:01 2024-09-16T03:37:21.190 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:21.257 INFO:teuthology.orchestra.run.smithi119.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 202 kB/s | 250 kB 00:01 2024-09-16T03:37:21.265 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T03:37:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T03:37:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T03:37:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T03:37:21.405 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:21.430 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:21.431 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:21.491 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T03:37:21.621 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T03:37:21.714 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T03:37:21.766 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T03:37:21.794 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:37:21.826 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T03:37:21.862 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:21.862 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi119.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi119.stdout:Total 8.4 MB/s | 38 MB 00:04 2024-09-16T03:37:22.553 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:22.580 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:22.580 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T03:37:22.766 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:37:22.866 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:37:22.877 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T03:37:22.902 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T03:37:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:37:23.034 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:23.034 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:23.087 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T03:37:23.109 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T03:37:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T03:37:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:23.860 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T03:37:23.862 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T03:37:23.906 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:37:23.981 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T03:37:24.007 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:37:24.049 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T03:37:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T03:37:24.264 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T03:37:24.316 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:37:24.316 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T03:37:24.316 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T03:37:24.316 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T03:37:24.316 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T03:37:24.316 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T03:37:24.317 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T03:37:24.317 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T03:37:24.317 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T03:37:24.317 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:37:24.317 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T03:37:24.317 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T03:37:24.343 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T03:37:24.638 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T03:37:24.638 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:24.638 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:24.638 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T03:37:24.638 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:24.638 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:24.638 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:24.639 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T03:37:24.639 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T03:37:24.639 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T03:37:24.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T03:37:24.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T03:37:24.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T03:37:24.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T03:37:24.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T03:37:24.639 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:24.639 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:24.869 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T03:37:25.004 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T03:37:25.127 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:37:25.427 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:12 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:25.560 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:25.561 INFO:teuthology.orchestra.run.smithi103.stdout:====================================================================================== 2024-09-16T03:37:25.561 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T03:37:25.561 INFO:teuthology.orchestra.run.smithi103.stdout:====================================================================================== 2024-09-16T03:37:25.561 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:37:25.561 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T03:37:25.561 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T03:37:25.562 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T03:37:25.563 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:25.563 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:37:25.563 INFO:teuthology.orchestra.run.smithi103.stdout:====================================================================================== 2024-09-16T03:37:25.563 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-09-16T03:37:25.563 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:25.563 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 38 M 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 212 M 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:37:25.999 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 33 kB 00:00 2024-09-16T03:37:26.200 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 362 kB/s | 18 kB 00:00 2024-09-16T03:37:26.258 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T03:37:26.284 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:37:26.300 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): ceph-mgr-diskprediction-local-19.1.1-32 16 MB/s | 7.4 MB 00:00 2024-09-16T03:37:26.325 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 307 kB/s | 38 kB 00:00 2024-09-16T03:37:26.375 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:37:26.434 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.1 MB/s | 3.0 MB 00:00 2024-09-16T03:37:26.676 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-16T03:37:26.752 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-09-16T03:37:26.910 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 8.1 MB/s | 4.9 MB 00:00 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi103.stdout:Total 24 MB/s | 38 MB 00:01 2024-09-16T03:37:27.399 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T03:37:27.430 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:37:27.516 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:27.542 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:27.542 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:27.980 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:27.981 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:28.739 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T03:37:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:37:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T03:37:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:37:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:37:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T03:37:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T03:37:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T03:37:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T03:37:28.865 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T03:37:28.865 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T03:37:28.865 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:37:28.865 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T03:37:28.867 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:37:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:37:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T03:37:29.187 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T03:37:29.217 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T03:37:29.295 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T03:37:29.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T03:37:29.303 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:29.303 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:29.303 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:29.303 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T03:37:29.303 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T03:37:29.303 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T03:37:29.303 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T03:37:29.303 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T03:37:29.303 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T03:37:29.304 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T03:37:29.304 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T03:37:29.304 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T03:37:29.304 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T03:37:29.304 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T03:37:29.304 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:29.304 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:29.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-09-16T03:37:29.955 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T03:37:30.209 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:17 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:30.343 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T03:37:30.344 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-09-16T03:37:30.345 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:30.346 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.2 M 2024-09-16T03:37:30.346 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-09-16T03:37:30.346 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T03:37:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:30.543 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T03:37:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T03:37:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T03:37:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T03:37:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T03:37:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T03:37:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T03:37:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T03:37:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T03:37:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T03:37:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T03:37:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:30.545 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 350 kB/s | 26 kB 00:00 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 195 kB/s | 19 kB 00:00 2024-09-16T03:37:30.694 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-09-16T03:37:30.720 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 258 kB/s | 49 kB 00:00 2024-09-16T03:37:30.745 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-16T03:37:30.770 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-09-16T03:37:30.779 INFO:teuthology.orchestra.run.smithi076.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.5 MB/s | 16 MB 00:10 2024-09-16T03:37:30.780 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:30.781 INFO:teuthology.orchestra.run.smithi076.stdout:Total 3.0 MB/s | 38 MB 00:12 2024-09-16T03:37:30.804 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-09-16T03:37:30.837 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-09-16T03:37:30.871 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 149 kB/s | 14 kB 00:00 2024-09-16T03:37:30.905 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-16T03:37:30.930 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-16T03:37:30.955 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.5 MB/s | 220 kB 00:00 2024-09-16T03:37:30.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-rook 2024-09-16T03:37:30.989 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-16T03:37:30.991 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:30.991 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-16T03:37:31.055 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:37:31.544 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:20 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi119.stdout:Install 13 Packages 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 2.2 M 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 26 M 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:31.837 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T03:37:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T03:37:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T03:37:31.982 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T03:37:32.005 INFO:teuthology.orchestra.run.smithi119.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 168 kB/s | 19 kB 00:00 2024-09-16T03:37:32.024 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T03:37:32.030 INFO:teuthology.orchestra.run.smithi119.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 184 kB/s | 26 kB 00:00 2024-09-16T03:37:32.056 INFO:teuthology.orchestra.run.smithi119.stdout:(3/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 299 kB/s | 49 kB 00:00 2024-09-16T03:37:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi119.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi119.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi119.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi119.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi119.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 350 kB/s | 32 kB 00:00 2024-09-16T03:37:32.283 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T03:37:32.299 INFO:teuthology.orchestra.run.smithi119.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-16T03:37:32.333 INFO:teuthology.orchestra.run.smithi119.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-16T03:37:32.359 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:37:32.366 INFO:teuthology.orchestra.run.smithi119.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-16T03:37:32.373 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T03:37:32.391 INFO:teuthology.orchestra.run.smithi119.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.5 MB/s | 220 kB 00:00 2024-09-16T03:37:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:37:32.425 INFO:teuthology.orchestra.run.smithi119.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-09-16T03:37:32.428 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:32.428 INFO:teuthology.orchestra.run.smithi119.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-09-16T03:37:32.443 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T03:37:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:37:32.494 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:32.510 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:32.510 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:32.587 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:37:32.601 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:32.603 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:32.646 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T03:37:32.830 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T03:37:32.861 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T03:37:32.937 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T03:37:32.996 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T03:37:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T03:37:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T03:37:33.111 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:33.147 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T03:37:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:37:33.271 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T03:37:33.326 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T03:37:33.390 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T03:37:33.427 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T03:37:33.468 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T03:37:33.571 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T03:37:33.596 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T03:37:33.683 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T03:37:33.725 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T03:37:33.846 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T03:37:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T03:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T03:37:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T03:37:34.266 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T03:37:34.266 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T03:37:34.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T03:37:34.268 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T03:37:34.268 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:34.268 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:34.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T03:37:34.566 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T03:37:34.618 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T03:37:34.651 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T03:37:34.700 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T03:37:34.738 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:37:34.989 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:37:35.058 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:22 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:37:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T03:37:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:37:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:37:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T03:37:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T03:37:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T03:37:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T03:37:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T03:37:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T03:37:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:37:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T03:37:35.192 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:35.193 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-16T03:37:35.193 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:35.193 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T03:37:35.193 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-16T03:37:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T03:37:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T03:37:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T03:37:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T03:37:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T03:37:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:35.194 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:35.194 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:35.194 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-09-16T03:37:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:35.195 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 7.0 M 2024-09-16T03:37:35.195 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 33 M 2024-09-16T03:37:35.195 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:35.448 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T03:37:35.449 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T03:37:35.564 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T03:37:35.564 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T03:37:35.565 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T03:37:35.566 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T03:37:35.566 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:35.566 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:35.850 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-16T03:37:35.875 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 685 kB/s | 146 kB 00:00 2024-09-16T03:37:35.934 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 991 kB/s | 58 kB 00:00 2024-09-16T03:37:35.959 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T03:37:35.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-rook 2024-09-16T03:37:35.992 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.8 MB/s | 548 kB 00:00 2024-09-16T03:37:36.024 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T03:37:36.052 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:37:36.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T03:37:36.101 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-16T03:37:36.103 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:36.103 INFO:teuthology.orchestra.run.smithi121.stdout:Total 7.7 MB/s | 7.0 MB 00:00 2024-09-16T03:37:36.174 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:36.285 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:36.285 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:36.548 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:23 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:36.631 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:36.684 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:36.685 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:36.685 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T03:37:36.685 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:36.685 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:37:36.685 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T03:37:36.685 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T03:37:36.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T03:37:36.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T03:37:36.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout:Install 13 Packages 2024-09-16T03:37:36.686 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:36.687 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.2 M 2024-09-16T03:37:36.687 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 26 M 2024-09-16T03:37:36.687 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:37:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T03:37:36.766 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:36.769 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 7.0 M 2024-09-16T03:37:36.769 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 33 M 2024-09-16T03:37:36.769 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:36.819 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T03:37:37.136 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T03:37:37.161 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 661 kB/s | 146 kB 00:00 2024-09-16T03:37:37.211 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-16T03:37:37.236 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T03:37:37.278 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.8 MB/s | 548 kB 00:00 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi119.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi103.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 197 kB/s | 19 kB 00:00 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T03:37:37.435 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:37.438 INFO:teuthology.orchestra.run.smithi103.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 196 kB/s | 26 kB 00:00 2024-09-16T03:37:37.446 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:37.446 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:37.486 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T03:37:37.489 INFO:teuthology.orchestra.run.smithi103.stdout:(3/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 272 kB/s | 49 kB 00:00 2024-09-16T03:37:37.515 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:37:37.523 INFO:teuthology.orchestra.run.smithi103.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-16T03:37:37.545 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:37.547 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:37.573 INFO:teuthology.orchestra.run.smithi103.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-16T03:37:37.607 INFO:teuthology.orchestra.run.smithi103.stdout:(6/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 946 kB/s | 159 kB 00:00 2024-09-16T03:37:37.649 INFO:teuthology.orchestra.run.smithi103.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 427 kB/s | 32 kB 00:00 2024-09-16T03:37:37.674 INFO:teuthology.orchestra.run.smithi103.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 210 kB/s | 14 kB 00:00 2024-09-16T03:37:37.699 INFO:teuthology.orchestra.run.smithi103.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-16T03:37:37.733 INFO:teuthology.orchestra.run.smithi103.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-16T03:37:37.767 INFO:teuthology.orchestra.run.smithi103.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 880 kB/s | 59 kB 00:00 2024-09-16T03:37:37.792 INFO:teuthology.orchestra.run.smithi103.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-16T03:37:37.841 INFO:teuthology.orchestra.run.smithi103.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-16T03:37:37.844 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:37.844 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-09-16T03:37:37.855 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:37.911 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:37.926 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:37.926 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:38.014 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:38.015 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T03:37:38.072 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T03:37:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:37:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T03:37:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T03:37:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T03:37:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T03:37:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T03:37:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:38.572 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T03:37:38.572 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:38.572 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:38.573 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:38.573 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T03:37:38.573 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T03:37:38.573 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T03:37:38.573 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T03:37:38.573 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T03:37:38.573 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:38.573 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:38.586 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T03:37:38.636 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:37:38.637 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T03:37:38.637 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:37:38.637 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:37:38.637 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T03:37:38.637 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T03:37:38.637 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T03:37:38.637 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T03:37:38.637 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T03:37:38.637 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T03:37:38.637 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:37:38.638 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T03:37:38.675 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T03:37:38.717 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T03:37:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T03:37:38.768 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T03:37:38.775 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:37:38.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-09-16T03:37:38.833 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T03:37:38.870 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T03:37:38.912 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T03:37:39.015 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T03:37:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T03:37:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:39.056 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:37:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T03:37:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T03:37:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T03:37:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T03:37:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T03:37:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T03:37:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T03:37:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T03:37:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T03:37:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T03:37:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T03:37:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:39.057 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:39.118 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T03:37:39.181 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T03:37:39.389 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:39.464 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-rook 2024-09-16T03:37:39.525 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:39.525 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:39.525 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-16T03:37:39.525 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:39.526 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:39.526 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T03:37:39.526 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-16T03:37:39.526 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T03:37:39.526 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:39.526 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:39.526 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:39.526 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-09-16T03:37:39.527 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:39.527 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 976 k 2024-09-16T03:37:39.527 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.9 M 2024-09-16T03:37:39.527 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:39.588 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:37:39.588 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T03:37:39.588 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T03:37:39.588 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T03:37:39.588 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T03:37:39.588 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T03:37:39.694 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 540 kB/s | 80 kB 00:00 2024-09-16T03:37:39.828 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.1 MB/s | 896 kB 00:00 2024-09-16T03:37:39.828 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:39.828 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.1 MB/s | 976 kB 00:00 2024-09-16T03:37:39.840 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:39.863 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:39.863 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:39.866 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:39.913 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T03:37:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:39.961 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T03:37:39.995 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T03:37:40.043 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T03:37:40.043 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:27 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:40.065 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:37:40.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-fuse 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi076.stdout:Install 13 Packages 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 2.2 M 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 26 M 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:37:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T03:37:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:37:40.687 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:40.754 INFO:teuthology.orchestra.run.smithi076.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 298 kB/s | 26 kB 00:00 2024-09-16T03:37:40.779 INFO:teuthology.orchestra.run.smithi076.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 172 kB/s | 19 kB 00:00 2024-09-16T03:37:40.820 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:40.821 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-16T03:37:40.821 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:40.821 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T03:37:40.821 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-16T03:37:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T03:37:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:40.822 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:40.822 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:40.822 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-09-16T03:37:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:40.822 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 976 k 2024-09-16T03:37:40.822 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 2.9 M 2024-09-16T03:37:40.822 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:40.846 INFO:teuthology.orchestra.run.smithi076.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-09-16T03:37:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:37:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T03:37:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T03:37:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T03:37:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T03:37:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T03:37:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T03:37:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T03:37:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T03:37:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T03:37:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T03:37:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T03:37:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T03:37:40.879 INFO:teuthology.orchestra.run.smithi076.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-16T03:37:40.897 INFO:teuthology.orchestra.run.smithi076.stdout:(5/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 215 kB/s | 49 kB 00:00 2024-09-16T03:37:40.914 INFO:teuthology.orchestra.run.smithi076.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.6 MB/s | 54 kB 00:00 2024-09-16T03:37:40.956 INFO:teuthology.orchestra.run.smithi076.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-09-16T03:37:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:37:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T03:37:41.023 INFO:teuthology.orchestra.run.smithi076.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 257 kB/s | 32 kB 00:00 2024-09-16T03:37:41.039 INFO:teuthology.orchestra.run.smithi076.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 112 kB/s | 14 kB 00:00 2024-09-16T03:37:41.073 INFO:teuthology.orchestra.run.smithi076.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-16T03:37:41.098 INFO:teuthology.orchestra.run.smithi076.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.7 MB/s | 220 kB 00:00 2024-09-16T03:37:41.124 INFO:teuthology.orchestra.run.smithi076.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.0 MB/s | 896 kB 00:00 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi076.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.5 MB/s | 1.0 MB 00:00 2024-09-16T03:37:41.186 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:41.186 INFO:teuthology.orchestra.run.smithi076.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-09-16T03:37:41.212 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:41.254 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:37:41.270 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:37:41.270 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:37:41.358 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:37:41.360 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:37:41.486 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T03:37:41.497 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 127 kB/s | 80 kB 00:00 2024-09-16T03:37:41.497 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:41.498 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.4 MB/s | 976 kB 00:00 2024-09-16T03:37:41.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-volume 2024-09-16T03:37:41.511 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:41.535 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:41.535 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:41.619 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:41.619 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:41.816 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:37:41.958 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T03:37:42.030 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:37:42.033 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T03:37:42.059 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:42.093 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:42.100 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T03:37:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 7.0 M 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 33 M 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 285 k 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.3 M 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T03:37:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T03:37:42.414 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-09-16T03:37:42.455 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T03:37:42.489 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.2 MB/s | 548 kB 00:00 2024-09-16T03:37:42.503 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T03:37:42.514 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-16T03:37:42.540 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T03:37:42.623 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-16T03:37:42.704 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:37:42.705 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T03:37:42.708 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 603 kB/s | 285 kB 00:00 2024-09-16T03:37:42.708 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:42.709 INFO:teuthology.orchestra.run.smithi121.stdout:Total 601 kB/s | 285 kB 00:00 2024-09-16T03:37:42.709 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:42.717 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:42.717 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:42.732 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 301 kB/s | 146 kB 00:00 2024-09-16T03:37:42.734 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:42.734 INFO:teuthology.orchestra.run.smithi103.stdout:Total 13 MB/s | 7.0 MB 00:00 2024-09-16T03:37:42.752 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:42.752 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:42.805 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:42.816 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:42.816 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:42.917 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:42.917 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:42.975 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:43.013 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T03:37:43.013 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:43.013 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:43.013 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T03:37:43.013 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:43.014 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:43.051 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:43.068 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:43.068 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:43.068 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T03:37:43.069 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:43.229 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:43.246 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T03:37:43.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-volume 2024-09-16T03:37:43.300 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T03:37:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T03:37:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T03:37:43.402 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T03:37:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:37:43.451 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T03:37:43.840 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:43.975 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:43.976 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:43.976 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-16T03:37:43.976 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:43.976 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:43.977 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T03:37:43.977 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:43.977 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:43.977 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:43.977 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-16T03:37:43.977 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:43.977 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 285 k 2024-09-16T03:37:43.977 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 1.3 M 2024-09-16T03:37:43.977 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:43.987 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T03:37:44.056 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:44.056 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:44.057 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:44.057 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:44.057 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:44.057 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:44.079 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T03:37:44.143 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T03:37:44.172 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:37:44.204 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:37:44.204 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T03:37:44.205 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T03:37:44.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-09-16T03:37:44.409 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 660 kB/s | 285 kB 00:00 2024-09-16T03:37:44.410 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:44.410 INFO:teuthology.orchestra.run.smithi119.stdout:Total 658 kB/s | 285 kB 00:00 2024-09-16T03:37:44.410 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:44.419 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:44.419 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:44.451 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:44.451 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T03:37:44.562 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T03:37:44.563 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T03:37:44.563 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T03:37:44.563 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T03:37:44.563 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T03:37:44.563 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T03:37:44.563 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:44.563 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:44.677 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:44.745 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:44.761 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:44.761 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:44.761 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T03:37:44.761 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:44.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T03:37:44.914 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T03:37:45.052 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:45.053 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:45.053 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-16T03:37:45.053 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:45.053 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:45.053 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T03:37:45.053 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:45.053 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:45.053 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:45.053 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-16T03:37:45.054 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:45.054 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 127 k 2024-09-16T03:37:45.054 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 456 k 2024-09-16T03:37:45.054 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:45.244 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T03:37:45.244 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:45.244 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:45.244 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:45.244 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T03:37:45.245 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T03:37:45.245 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T03:37:45.245 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T03:37:45.245 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T03:37:45.245 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:45.245 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 370 kB/s | 127 kB 00:00 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi121.stdout:Total 368 kB/s | 127 kB 00:00 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:45.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse 2024-09-16T03:37:45.555 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:45.556 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:45.556 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T03:37:45.556 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:45.556 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:37:45.556 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout:Install 6 Packages 2024-09-16T03:37:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:45.558 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 7.0 M 2024-09-16T03:37:45.558 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 33 M 2024-09-16T03:37:45.558 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:37:45.642 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:45.795 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:45.795 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:45.795 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:45.795 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:45.796 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:45.796 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:46.059 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:46.094 INFO:teuthology.orchestra.run.smithi076.stdout:(1/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 671 kB/s | 146 kB 00:00 2024-09-16T03:37:46.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados-devel 2024-09-16T03:37:46.144 INFO:teuthology.orchestra.run.smithi076.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-16T03:37:46.178 INFO:teuthology.orchestra.run.smithi076.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-09-16T03:37:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:46.193 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:46.194 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T03:37:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:37:46.194 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T03:37:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T03:37:46.194 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T03:37:46.194 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:37:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-09-16T03:37:46.195 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 976 k 2024-09-16T03:37:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 2.9 M 2024-09-16T03:37:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:37:46.203 INFO:teuthology.orchestra.run.smithi076.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T03:37:46.463 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:46.464 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:46.464 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:46.464 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:46.464 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:46.464 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:46.503 INFO:teuthology.orchestra.run.smithi076.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 396 kB/s | 248 kB 00:00 2024-09-16T03:37:46.680 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:46.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-09-16T03:37:46.818 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:46.818 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 127 k 2024-09-16T03:37:46.819 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 456 k 2024-09-16T03:37:46.820 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:47.018 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 1.6 MB/s | 896 kB 00:00 2024-09-16T03:37:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 122 kB/s | 80 kB 00:00 2024-09-16T03:37:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.0 MB/s | 976 kB 00:00 2024-09-16T03:37:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:47.164 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:47.164 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:47.207 INFO:teuthology.orchestra.run.smithi119.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 328 kB/s | 127 kB 00:00 2024-09-16T03:37:47.207 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:47.207 INFO:teuthology.orchestra.run.smithi119.stdout:Total 326 kB/s | 127 kB 00:00 2024-09-16T03:37:47.208 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:47.213 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:47.213 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:47.364 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:47.409 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:47.410 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-16T03:37:47.410 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:47.425 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:47.459 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:47.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-09-16T03:37:47.556 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:47.571 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T03:37:47.634 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:37:47.978 INFO:teuthology.orchestra.run.smithi076.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.8 MB/s | 6.0 MB 00:02 2024-09-16T03:37:47.980 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:47.981 INFO:teuthology.orchestra.run.smithi076.stdout:Total 2.9 MB/s | 7.0 MB 00:02 2024-09-16T03:37:48.003 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:48.038 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:48.054 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:37:48.065 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:37:48.066 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:37:48.166 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:37:48.167 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:37:48.174 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:48.174 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:48.174 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-16T03:37:48.174 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:48.174 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:48.174 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T03:37:48.174 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:48.175 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:48.175 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:48.175 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-16T03:37:48.175 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:48.175 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 32 k 2024-09-16T03:37:48.175 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 142 k 2024-09-16T03:37:48.175 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:48.230 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:48.231 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:48.231 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:48.231 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:48.231 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:48.231 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:37:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T03:37:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:37:48.483 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 103 kB/s | 32 kB 00:00 2024-09-16T03:37:48.483 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:48.484 INFO:teuthology.orchestra.run.smithi121.stdout:Total 103 kB/s | 32 kB 00:00 2024-09-16T03:37:48.484 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:48.487 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:48.488 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:48.505 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:48.505 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:48.528 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs2 2024-09-16T03:37:48.592 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T03:37:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:48.628 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T03:37:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:48.628 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:48.645 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T03:37:48.688 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:48.697 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T03:37:48.933 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-volume 2024-09-16T03:37:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:49.118 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:49.213 INFO:teuthology.orchestra.run.smithi119.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:49.226 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T03:37:49.257 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:49.258 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-16T03:37:49.258 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:49.304 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:49.304 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:49.316 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T03:37:49.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs-devel 2024-09-16T03:37:49.380 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T03:37:49.409 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:37:49.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-09-16T03:37:49.510 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:49.650 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:49.651 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:49.651 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T03:37:49.651 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:49.651 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:37:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T03:37:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:49.652 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:37:49.652 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:49.652 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T03:37:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:49.652 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 285 k 2024-09-16T03:37:49.652 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.3 M 2024-09-16T03:37:49.652 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:37:49.888 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:49.933 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.0 MB/s | 285 kB 00:00 2024-09-16T03:37:49.934 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:49.934 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.0 MB/s | 285 kB 00:00 2024-09-16T03:37:49.934 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:49.943 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:49.943 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:50.020 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:50.021 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:50.021 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-16T03:37:50.021 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:50.021 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:50.021 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T03:37:50.021 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:50.022 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:50.022 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:50.022 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-16T03:37:50.022 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:50.022 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 32 k 2024-09-16T03:37:50.022 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 142 k 2024-09-16T03:37:50.022 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:50.039 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:50.129 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi119.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 204 kB/s | 32 kB 00:00 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi119.stdout:Total 203 kB/s | 32 kB 00:00 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:50.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-09-16T03:37:50.260 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:50.277 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:50.277 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:50.277 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T03:37:50.277 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:50.296 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:50.399 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T03:37:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:50.493 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:37:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T03:37:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T03:37:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T03:37:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T03:37:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T03:37:50.494 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:50.494 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:50.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-fuse 2024-09-16T03:37:50.801 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:50.810 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:50.890 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:50.933 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:50.934 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-16T03:37:50.934 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:51.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-09-16T03:37:51.048 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:51.048 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:51.048 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:51.048 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:51.048 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:51.048 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:51.233 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados2 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:51.468 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:51.468 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T03:37:51.469 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:51.469 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:37:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T03:37:51.469 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T03:37:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T03:37:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:51.469 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:37:51.470 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:51.470 INFO:teuthology.orchestra.run.smithi076.stdout:Install 2 Packages 2024-09-16T03:37:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:51.470 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 976 k 2024-09-16T03:37:51.470 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 2.9 M 2024-09-16T03:37:51.470 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:37:51.552 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel 2024-09-16T03:37:51.588 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:51.678 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:51.721 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:51.722 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-16T03:37:51.722 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:51.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi076.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 533 kB/s | 80 kB 00:00 2024-09-16T03:37:51.933 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:51.934 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-16T03:37:51.934 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:52.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librbd1 2024-09-16T03:37:52.039 INFO:teuthology.orchestra.run.smithi076.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.0 MB/s | 896 kB 00:00 2024-09-16T03:37:52.040 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:52.041 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.7 MB/s | 976 kB 00:00 2024-09-16T03:37:52.051 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:37:52.075 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:37:52.075 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:37:52.131 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:52.156 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:37:52.156 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:37:52.269 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:52.270 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:52.270 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-16T03:37:52.270 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:52.270 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:37:52.270 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T03:37:52.270 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:52.270 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:37:52.270 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:52.271 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T03:37:52.271 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:52.271 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 127 k 2024-09-16T03:37:52.271 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 456 k 2024-09-16T03:37:52.271 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:37:52.343 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:37:52.354 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:52.444 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 624 kB/s | 127 kB 00:00 2024-09-16T03:37:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:Total 619 kB/s | 127 kB 00:00 2024-09-16T03:37:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:52.482 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T03:37:52.487 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:52.488 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-16T03:37:52.488 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:52.551 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:52.551 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:52.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-09-16T03:37:52.554 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:37:52.568 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:52.660 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:52.703 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:52.704 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-16T03:37:52.704 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:52.717 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:52.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rados 2024-09-16T03:37:52.813 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:53.116 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:53.207 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:53.241 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:53.250 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:53.251 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-16T03:37:53.251 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:53.270 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:37:53.270 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T03:37:53.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-09-16T03:37:53.350 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:53.441 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:53.484 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:53.485 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-16T03:37:53.485 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:53.535 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:53.535 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:53.535 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:53.535 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:53.535 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:53.535 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:53.540 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T03:37:53.541 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:53.541 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:37:53.541 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T03:37:53.541 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:53.541 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:53.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rgw 2024-09-16T03:37:53.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2 2024-09-16T03:37:53.831 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-volume 2024-09-16T03:37:53.881 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:53.972 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:54.015 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:54.016 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-16T03:37:54.016 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:54.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-09-16T03:37:54.119 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:54.211 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:54.255 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:54.256 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-16T03:37:54.256 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:54.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-cephfs 2024-09-16T03:37:54.339 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:54.409 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:54.433 INFO:teuthology.orchestra.run.smithi103.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:54.475 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:54.476 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T03:37:54.477 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:54.549 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs-devel 2024-09-16T03:37:54.550 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:54.550 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:54.550 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T03:37:54.550 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:54.550 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:37:54.550 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T03:37:54.550 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:54.551 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:37:54.551 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:54.551 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T03:37:54.551 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:54.551 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 285 k 2024-09-16T03:37:54.551 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 1.3 M 2024-09-16T03:37:54.551 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:37:54.653 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:54.790 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:54.790 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 85 k 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 231 k 2024-09-16T03:37:54.791 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:54.889 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:54.919 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 774 kB/s | 285 kB 00:00 2024-09-16T03:37:54.920 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:54.920 INFO:teuthology.orchestra.run.smithi076.stdout:Total 771 kB/s | 285 kB 00:00 2024-09-16T03:37:54.920 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:37:54.929 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:37:54.929 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:37:54.954 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:37:54.955 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 437 kB/s | 85 kB 00:00 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi121.stdout:Total 434 kB/s | 85 kB 00:00 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:55.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rbd 2024-09-16T03:37:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:55.180 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:37:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:55.252 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-16T03:37:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:37:55.252 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T03:37:55.252 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:37:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:37:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T03:37:55.253 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 32 k 2024-09-16T03:37:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 142 k 2024-09-16T03:37:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:37:55.255 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:55.271 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:55.272 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:55.272 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T03:37:55.272 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 198 kB/s | 32 kB 00:00 2024-09-16T03:37:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:Total 196 kB/s | 32 kB 00:00 2024-09-16T03:37:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:37:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:37:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:37:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:37:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:37:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:55.659 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:55.751 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:55.795 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:55.796 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-16T03:37:55.796 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:55.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-fuse 2024-09-16T03:37:55.983 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:55.983 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:55.983 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:55.983 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:56.053 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:56.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-09-16T03:37:56.283 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:37:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:56.284 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:37:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:37:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:56.319 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:37:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:37:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:56.319 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:56.427 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:56.453 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 85 k 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 231 k 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:56.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados-devel 2024-09-16T03:37:56.742 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 477 kB/s | 85 kB 00:00 2024-09-16T03:37:56.742 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:56.743 INFO:teuthology.orchestra.run.smithi119.stdout:Total 474 kB/s | 85 kB 00:00 2024-09-16T03:37:56.743 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:56.750 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:56.750 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:56.796 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:56.815 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:56.815 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:56.940 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:37:56.941 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:56.941 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-16T03:37:56.941 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:56.941 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:37:56.941 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T03:37:56.941 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:56.941 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:37:56.942 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:37:56.942 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-16T03:37:56.942 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:56.942 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.1 M 2024-09-16T03:37:56.942 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 13 M 2024-09-16T03:37:56.942 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T03:37:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:57.201 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:57.220 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1 2024-09-16T03:37:57.332 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 8.0 MB/s | 3.1 MB 00:00 2024-09-16T03:37:57.332 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:57.333 INFO:teuthology.orchestra.run.smithi121.stdout:Total 7.9 MB/s | 3.1 MB 00:00 2024-09-16T03:37:57.333 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:37:57.338 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:57.340 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:57.340 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-09-16T03:37:57.340 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:57.340 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:37:57.340 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T03:37:57.340 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:57.341 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:37:57.341 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:37:57.341 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T03:37:57.341 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:57.341 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 127 k 2024-09-16T03:37:57.341 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 456 k 2024-09-16T03:37:57.341 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:37:57.342 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:37:57.342 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:37:57.425 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:37:57.425 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:37:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:57.542 INFO:teuthology.orchestra.run.smithi076.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 627 kB/s | 127 kB 00:00 2024-09-16T03:37:57.543 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:57.543 INFO:teuthology.orchestra.run.smithi076.stdout:Total 622 kB/s | 127 kB 00:00 2024-09-16T03:37:57.543 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:37:57.549 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:37:57.549 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:37:57.619 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:37:57.619 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:37:57.711 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:57.712 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:57.712 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:37:57.712 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:57.712 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:57.712 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:37:57.782 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:37:57.785 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:57.873 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:57.873 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:37:57.875 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:57.919 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:57.920 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T03:37:57.920 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:57.932 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:57.958 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:57.958 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T03:37:57.959 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:37:57.959 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:37:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:57.991 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-mirror 2024-09-16T03:37:57.992 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rados 2024-09-16T03:37:58.339 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:58.560 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:58.570 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:37:58.627 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:58.696 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:37:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:37:58.696 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:58.697 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:58.697 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:58.697 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T03:37:58.698 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:58.707 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:37:58.707 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:58.707 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-16T03:37:58.708 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:58.708 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:37:58.708 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T03:37:58.708 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:58.708 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:37:58.708 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:37:58.708 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-16T03:37:58.708 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:58.709 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.1 M 2024-09-16T03:37:58.709 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 13 M 2024-09-16T03:37:58.709 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:37:58.790 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rgw 2024-09-16T03:37:58.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs2 2024-09-16T03:37:59.051 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 9.1 MB/s | 3.1 MB 00:00 2024-09-16T03:37:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:37:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-09-16T03:37:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:37:59.061 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:37:59.061 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:37:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:59.084 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:37:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:37:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:37:59.085 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:37:59.143 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:37:59.143 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:37:59.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-09-16T03:37:59.355 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:59.444 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:59.464 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:37:59.488 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:37:59.493 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T03:37:59.493 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:37:59.557 INFO:teuthology.orchestra.run.smithi076.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:37:59.563 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-cephfs 2024-09-16T03:37:59.582 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:37:59.601 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:37:59.601 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T03:37:59.602 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:37:59.643 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:59.669 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:37:59.669 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:37:59.669 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T03:37:59.669 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:37:59.669 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:37:59.669 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:37:59.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs-devel 2024-09-16T03:37:59.912 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:00.045 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-16T03:38:00.046 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:38:00.046 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-16T03:38:00.046 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:38:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-16T03:38:00.047 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T03:38:00.047 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-16T03:38:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-16T03:38:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-16T03:38:00.047 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 170 k 2024-09-16T03:38:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 494 k 2024-09-16T03:38:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-16T03:38:00.133 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 800 kB/s | 170 kB 00:00 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi121.stdout:Total 794 kB/s | 170 kB 00:00 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-16T03:38:00.268 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:38:00.334 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-16T03:38:00.334 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-16T03:38:00.347 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rbd 2024-09-16T03:38:00.395 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:38:00.395 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:00.395 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-09-16T03:38:00.395 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:00.395 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:38:00.395 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T03:38:00.395 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:00.396 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:38:00.396 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:00.396 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T03:38:00.396 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:00.396 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 32 k 2024-09-16T03:38:00.396 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 142 k 2024-09-16T03:38:00.396 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:38:00.487 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-16T03:38:00.539 INFO:teuthology.orchestra.run.smithi076.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 222 kB/s | 32 kB 00:00 2024-09-16T03:38:00.539 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:38:00.540 INFO:teuthology.orchestra.run.smithi076.stdout:Total 219 kB/s | 32 kB 00:00 2024-09-16T03:38:00.540 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:38:00.543 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:38:00.543 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:38:00.566 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:38:00.566 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:38:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:38:00.761 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:00.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:00.836 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:00.836 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:38:00.836 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:38:00.836 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:00.836 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:38:00.911 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:01.001 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:01.001 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:38:01.046 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T03:38:01.046 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:38:01.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-nbd 2024-09-16T03:38:01.111 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:01.262 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:01.262 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:01.262 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-16T03:38:01.262 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:38:01.262 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:01.262 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-16T03:38:01.392 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:01.392 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:01.392 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:38:01.392 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:38:01.392 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:01.392 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:38:01.489 DEBUG:teuthology.parallel:result is None 2024-09-16T03:38:01.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados2 2024-09-16T03:38:01.662 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 03:36:11 AM UTC. 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:01.802 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:01.803 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 170 k 2024-09-16T03:38:01.804 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 494 k 2024-09-16T03:38:01.804 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-16T03:38:01.812 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:38:01.812 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:38:01.812 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 85 k 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 231 k 2024-09-16T03:38:01.813 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:38:02.000 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 456 kB/s | 85 kB 00:00 2024-09-16T03:38:02.001 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:38:02.001 INFO:teuthology.orchestra.run.smithi103.stdout:Total 452 kB/s | 85 kB 00:00 2024-09-16T03:38:02.001 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:38:02.008 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:38:02.008 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:38:02.013 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 814 kB/s | 170 kB 00:00 2024-09-16T03:38:02.014 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:38:02.014 INFO:teuthology.orchestra.run.smithi119.stdout:Total 808 kB/s | 170 kB 00:00 2024-09-16T03:38:02.014 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-16T03:38:02.022 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-16T03:38:02.022 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-16T03:38:02.072 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:38:02.073 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:38:02.086 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-16T03:38:02.087 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-16T03:38:02.141 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi076.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:38:02.274 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T03:38:02.274 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:38:02.337 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:02.339 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:02.356 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librbd1 2024-09-16T03:38:02.699 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:02.742 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:02.919 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:02.985 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:38:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:38:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:02.985 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:38:03.005 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:03.005 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:03.005 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-16T03:38:03.005 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:38:03.005 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:03.005 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-16T03:38:03.010 INFO:teuthology.orchestra.run.smithi076.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:38:03.054 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:38:03.055 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T03:38:03.055 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:38:03.134 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rados 2024-09-16T03:38:03.242 DEBUG:teuthology.parallel:result is None 2024-09-16T03:38:03.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2024-09-16T03:38:03.703 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:03.794 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:38:03.839 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:38:03.840 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T03:38:03.840 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:38:03.856 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:03.923 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rgw 2024-09-16T03:38:03.999 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:38:03.999 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 3.1 M 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 13 M 2024-09-16T03:38:04.000 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:38:04.358 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 8.7 MB/s | 3.1 MB 00:00 2024-09-16T03:38:04.359 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:38:04.359 INFO:teuthology.orchestra.run.smithi103.stdout:Total 8.7 MB/s | 3.1 MB 00:00 2024-09-16T03:38:04.359 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:38:04.366 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:38:04.366 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:38:04.448 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:38:04.449 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:38:04.490 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:04.581 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:38:04.624 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:38:04.625 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T03:38:04.625 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:38:04.704 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-cephfs 2024-09-16T03:38:04.892 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:38:04.966 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:04.991 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:04.991 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:38:04.991 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T03:38:04.991 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:38:04.992 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:38:04.992 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:05.264 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:52 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:38:05.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rbd 2024-09-16T03:38:06.051 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:06.142 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:38:06.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:06.175 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:06.175 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:38:06.175 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:38:06.176 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:06.176 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:38:06.185 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:38:06.186 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T03:38:06.186 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:38:06.288 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-fuse 2024-09-16T03:38:06.403 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:06.986 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:07.013 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:38:07.013 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:07.013 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-09-16T03:38:07.013 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:07.014 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:38:07.014 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T03:38:07.014 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:07.014 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:38:07.014 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:07.014 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T03:38:07.014 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:07.014 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 85 k 2024-09-16T03:38:07.015 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 231 k 2024-09-16T03:38:07.015 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:38:07.125 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T03:38:07.125 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:38:07.125 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 170 k 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 494 k 2024-09-16T03:38:07.126 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T03:38:07.209 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 439 kB/s | 85 kB 00:00 2024-09-16T03:38:07.210 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:38:07.210 INFO:teuthology.orchestra.run.smithi076.stdout:Total 435 kB/s | 85 kB 00:00 2024-09-16T03:38:07.210 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:38:07.218 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:38:07.218 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:38:07.353 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 752 kB/s | 170 kB 00:00 2024-09-16T03:38:07.354 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:38:07.354 INFO:teuthology.orchestra.run.smithi103.stdout:Total 747 kB/s | 170 kB 00:00 2024-09-16T03:38:07.354 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T03:38:07.362 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T03:38:07.362 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T03:38:07.428 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:38:07.428 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T03:38:07.428 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T03:38:07.548 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:07.579 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T03:38:07.686 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:07.976 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:08.118 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:08.234 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:08.234 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:08.234 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:38:08.234 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:38:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:08.235 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:38:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:08.378 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T03:38:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:38:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:08.379 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T03:38:08.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-mirror 2024-09-16T03:38:08.584 DEBUG:teuthology.parallel:result is None 2024-09-16T03:38:09.093 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:56 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:09.232 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:38:09.232 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 3.1 M 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 13 M 2024-09-16T03:38:09.233 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:38:09.596 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 8.6 MB/s | 3.1 MB 00:00 2024-09-16T03:38:09.596 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:38:09.597 INFO:teuthology.orchestra.run.smithi076.stdout:Total 8.5 MB/s | 3.1 MB 00:00 2024-09-16T03:38:09.597 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:38:09.606 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:38:09.606 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:38:09.687 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:38:09.687 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:38:10.124 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:38:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:10.206 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:38:10.206 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T03:38:10.206 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:38:10.206 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:38:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:38:11.650 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-nbd 2024-09-16T03:38:12.228 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:59 ago on Mon 16 Sep 2024 03:36:13 AM UTC. 2024-09-16T03:38:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T03:38:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 170 k 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 494 k 2024-09-16T03:38:12.365 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T03:38:12.575 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 814 kB/s | 170 kB 00:00 2024-09-16T03:38:12.576 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:38:12.576 INFO:teuthology.orchestra.run.smithi076.stdout:Total 809 kB/s | 170 kB 00:00 2024-09-16T03:38:12.576 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T03:38:12.583 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T03:38:12.583 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T03:38:12.650 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T03:38:12.650 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T03:38:12.811 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T03:38:12.922 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:38:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:13.642 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T03:38:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:38:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:13.642 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T03:38:13.879 DEBUG:teuthology.parallel:result is None 2024-09-16T03:38:13.879 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:38:14.026 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T03:38:14.047 INFO:teuthology.orchestra.run.smithi076.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T03:38:14.048 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T03:38:14.048 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T03:38:14.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:38:14.188 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T03:38:14.213 INFO:teuthology.orchestra.run.smithi103.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T03:38:14.213 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T03:38:14.213 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T03:38:14.215 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:38:14.352 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T03:38:14.379 INFO:teuthology.orchestra.run.smithi119.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T03:38:14.379 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T03:38:14.379 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T03:38:14.381 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:38:14.519 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T03:38:14.546 INFO:teuthology.orchestra.run.smithi121.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T03:38:14.546 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T03:38:14.546 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T03:38:14.548 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T03:38:14.548 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:38:14.548 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:38:14.600 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:38:14.600 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:38:14.631 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:38:14.631 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:38:14.662 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:38:14.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:38:14.692 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T03:38:14.692 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:38:14.692 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:38:14.722 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:38:14.791 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:38:14.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:38:14.821 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:38:14.889 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:38:14.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:38:14.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:38:14.987 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:38:14.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:38:15.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:38:15.090 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T03:38:15.090 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:38:15.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:38:15.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:38:15.187 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:38:15.187 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:38:15.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:38:15.284 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:38:15.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:38:15.313 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:38:15.382 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:38:15.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:38:15.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:38:15.480 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T03:38:15.481 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:38:15.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:38:15.510 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:38:15.578 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:38:15.578 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:38:15.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:38:15.676 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:38:15.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:38:15.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:38:15.774 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:38:15.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:38:15.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:38:15.873 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T03:38:15.956 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T03:38:15.957 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /var/log/ceph 2024-09-16T03:38:15.959 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2024-09-16T03:38:15.962 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /var/log/ceph 2024-09-16T03:38:15.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2024-09-16T03:38:15.992 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T03:38:15.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:38:16.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:38:16.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:38:16.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:38:16.061 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T03:38:16.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:38:16.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:38:16.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:38:16.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:38:16.139 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T03:38:16.140 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T03:38:16.140 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894', 'branch': 'squid', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905894', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi076.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCNbHsjzHuQ2cUQigTd2Zn5fjtng3xHz/bfDQyfr/jeGKA1bXp39Tfl5sQ2PpbgVsy/ASTUOhdae+rotcKAQV38=', 'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFz+8VMv6RaQlzLP67KTQeG6p329gwORwDSlf+NDA4pPTK9wjtc80tTfi7oN+nmEpfW8EOxoRef50PsqFIqaTro=', 'smithi119.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJdfUzuR24onZlhr0YvfJhNnZDtOkcOLo7drXQ0vCtlRfm/D3py8sJcHIwSuWQBhQj87lC4719RLKwOSc03Sph0=', 'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDAv/poHrnNt/ymxP7JkuPVKBmUmzvhSBZyn04dchPsF9uDmufTWPJ/TfJrKboV+GN293qv+Gg8+vxDIiU9Ruaw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T03:38:16.140 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:38:16.164 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:38:16.170 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:38:16.172 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:38:16.188 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:38:16.221 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:38:16.228 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:38:16.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:38:16.256 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:38:16.256 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:38:16.302 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:38:16.302 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2024-09-16T03:38:16.359 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:38:16.359 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:16.359 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T03:38:16.359 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:16.359 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:16.359 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 03:38:10.469788970 +0000 2024-09-16T03:38:16.359 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 03:35:31.033845186 +0000 2024-09-16T03:38:16.359 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 03:35:31.033845186 +0000 2024-09-16T03:38:16.359 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-16 03:35:31.033845186 +0000 2024-09-16T03:38:16.360 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:38:16.424 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T03:38:16.424 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T03:38:16.424 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000207276 s, 2.5 MB/s 2024-09-16T03:38:16.426 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:38:16.482 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 03:38:10.469788970 +0000 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 03:35:31.369834562 +0000 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 03:35:31.369834562 +0000 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-16 03:35:31.369834562 +0000 2024-09-16T03:38:16.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:38:16.602 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T03:38:16.603 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T03:38:16.603 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000239627 s, 2.1 MB/s 2024-09-16T03:38:16.604 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:38:16.660 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 03:38:10.469788970 +0000 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 03:35:31.696824222 +0000 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 03:35:31.696824222 +0000 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-16 03:35:31.696824222 +0000 2024-09-16T03:38:16.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:38:16.779 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T03:38:16.780 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T03:38:16.780 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000202604 s, 2.5 MB/s 2024-09-16T03:38:16.781 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:38:16.837 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2024-09-16T03:38:16.893 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:38:16.893 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:16.894 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-16T03:38:16.894 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:16.894 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:16.894 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 03:38:10.469788970 +0000 2024-09-16T03:38:16.894 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 03:35:32.033813567 +0000 2024-09-16T03:38:16.894 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 03:35:32.033813567 +0000 2024-09-16T03:38:16.895 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-16 03:35:32.033813567 +0000 2024-09-16T03:38:16.895 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:38:16.956 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T03:38:16.957 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T03:38:16.957 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000157895 s, 3.2 MB/s 2024-09-16T03:38:16.958 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:38:17.014 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:17.014 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:38:17.014 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:38:17.034 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:38:17.035 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-16 03:38:05.236954352 +0000 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-16 03:35:31.019409296 +0000 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-16 03:35:31.019409296 +0000 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-16 03:35:31.019409296 +0000 2024-09-16T03:38:17.094 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:38:17.162 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-16T03:38:17.162 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-16T03:38:17.163 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.00022623 s, 2.3 MB/s 2024-09-16T03:38:17.164 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:38:17.224 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-09-16T03:38:17.282 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:38:17.282 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:17.282 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T03:38:17.282 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:17.282 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:17.282 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-16 03:38:05.237954334 +0000 2024-09-16T03:38:17.282 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-16 03:35:31.358402993 +0000 2024-09-16T03:38:17.282 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-16 03:35:31.358402993 +0000 2024-09-16T03:38:17.282 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-16 03:35:31.358402993 +0000 2024-09-16T03:38:17.283 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:38:17.349 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-16T03:38:17.349 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-16T03:38:17.349 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000198057 s, 2.6 MB/s 2024-09-16T03:38:17.351 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:38:17.408 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-09-16T03:38:17.465 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:38:17.465 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:17.465 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T03:38:17.465 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:17.465 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:17.465 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-16 03:38:05.237954334 +0000 2024-09-16T03:38:17.465 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-16 03:35:31.683396950 +0000 2024-09-16T03:38:17.466 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-16 03:35:31.683396950 +0000 2024-09-16T03:38:17.466 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-16 03:35:31.683396950 +0000 2024-09-16T03:38:17.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000198124 s, 2.6 MB/s 2024-09-16T03:38:17.532 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:38:17.589 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-09-16T03:38:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:38:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:17.646 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-16T03:38:17.646 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:17.646 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:17.646 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-16 03:38:05.237954334 +0000 2024-09-16T03:38:17.646 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-16 03:35:32.017390740 +0000 2024-09-16T03:38:17.646 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-16 03:35:32.017390740 +0000 2024-09-16T03:38:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-16 03:35:32.017390740 +0000 2024-09-16T03:38:17.646 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:38:17.709 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-16T03:38:17.710 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-16T03:38:17.710 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000199157 s, 2.6 MB/s 2024-09-16T03:38:17.711 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:38:17.768 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:17.768 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:38:17.768 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:38:17.787 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:38:17.787 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-09-16T03:38:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:38:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:17.847 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T03:38:17.847 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:17.847 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:17.847 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-16 03:37:59.938155874 +0000 2024-09-16T03:38:17.847 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-16 03:35:32.108049966 +0000 2024-09-16T03:38:17.847 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-16 03:35:32.108049966 +0000 2024-09-16T03:38:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-16 03:35:32.108049966 +0000 2024-09-16T03:38:17.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:38:17.915 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-16T03:38:17.915 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-16T03:38:17.915 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000298467 s, 1.7 MB/s 2024-09-16T03:38:17.916 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:38:17.975 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-09-16T03:38:18.033 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:38:18.033 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:18.033 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T03:38:18.033 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:18.033 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:18.033 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-16 03:37:59.938155874 +0000 2024-09-16T03:38:18.033 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-16 03:35:32.437042355 +0000 2024-09-16T03:38:18.034 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-16 03:35:32.437042355 +0000 2024-09-16T03:38:18.034 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-16 03:35:32.437042355 +0000 2024-09-16T03:38:18.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:38:18.100 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-16T03:38:18.100 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-16T03:38:18.100 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000237757 s, 2.2 MB/s 2024-09-16T03:38:18.101 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:38:18.159 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-09-16T03:38:18.216 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:38:18.216 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:18.216 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T03:38:18.216 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:18.216 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:18.217 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-16 03:37:59.939155850 +0000 2024-09-16T03:38:18.217 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-16 03:35:32.763034813 +0000 2024-09-16T03:38:18.217 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-16 03:35:32.763034813 +0000 2024-09-16T03:38:18.217 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-16 03:35:32.763034813 +0000 2024-09-16T03:38:18.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:38:18.283 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-16T03:38:18.283 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-16T03:38:18.283 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000223406 s, 2.3 MB/s 2024-09-16T03:38:18.284 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:38:18.342 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-09-16T03:38:18.397 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:38:18.397 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:18.397 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T03:38:18.397 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:18.398 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:18.398 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-16 03:37:59.939155850 +0000 2024-09-16T03:38:18.398 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-16 03:35:33.087027317 +0000 2024-09-16T03:38:18.398 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-16 03:35:33.087027317 +0000 2024-09-16T03:38:18.398 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-16 03:35:33.087027317 +0000 2024-09-16T03:38:18.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:38:18.461 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-16T03:38:18.461 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-16T03:38:18.461 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000234867 s, 2.2 MB/s 2024-09-16T03:38:18.462 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:38:18.519 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:18.519 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:38:18.519 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:38:18.538 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:38:18.538 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-09-16T03:38:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:38:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:18.597 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T03:38:18.597 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:18.597 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:18.598 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-16 03:37:58.228938954 +0000 2024-09-16T03:38:18.598 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-16 03:35:32.325559664 +0000 2024-09-16T03:38:18.598 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-16 03:35:32.325559664 +0000 2024-09-16T03:38:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-16 03:35:32.325559664 +0000 2024-09-16T03:38:18.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:38:18.664 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-16T03:38:18.664 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-16T03:38:18.664 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000348921 s, 1.5 MB/s 2024-09-16T03:38:18.666 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:38:18.725 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-09-16T03:38:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:38:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:18.782 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T03:38:18.782 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:18.782 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:18.783 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-16 03:37:58.228938954 +0000 2024-09-16T03:38:18.783 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-16 03:35:32.669555068 +0000 2024-09-16T03:38:18.783 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-16 03:35:32.669555068 +0000 2024-09-16T03:38:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-16 03:35:32.669555068 +0000 2024-09-16T03:38:18.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:38:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-16T03:38:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-16T03:38:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000261862 s, 2.0 MB/s 2024-09-16T03:38:18.850 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:38:18.908 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-09-16T03:38:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:38:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:18.965 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T03:38:18.965 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:18.965 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:18.965 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-16 03:37:58.229938941 +0000 2024-09-16T03:38:18.966 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-16 03:35:33.028550271 +0000 2024-09-16T03:38:18.966 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-16 03:35:33.028550271 +0000 2024-09-16T03:38:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-16 03:35:33.028550271 +0000 2024-09-16T03:38:18.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:38:19.030 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-16T03:38:19.030 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-16T03:38:19.030 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000182119 s, 2.8 MB/s 2024-09-16T03:38:19.031 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:38:19.088 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-09-16T03:38:19.144 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:38:19.144 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:38:19.144 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-16T03:38:19.144 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:38:19.145 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:38:19.145 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-16 03:37:58.229938941 +0000 2024-09-16T03:38:19.145 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-16 03:35:33.360545835 +0000 2024-09-16T03:38:19.145 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-16 03:35:33.360545835 +0000 2024-09-16T03:38:19.145 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-16 03:35:33.360545835 +0000 2024-09-16T03:38:19.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:38:19.208 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-16T03:38:19.208 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-16T03:38:19.209 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000219217 s, 2.3 MB/s 2024-09-16T03:38:19.210 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:38:19.267 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:19.267 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-16T03:38:19.267 INFO:tasks.ceph:Generating config... 2024-09-16T03:38:19.268 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T03:38:19.269 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.1 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T03:38:19.270 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T03:38:19.271 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-16T03:38:19.271 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-16T03:38:19.271 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T03:38:19.271 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T03:38:19.271 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-16T03:38:19.271 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-16T03:38:19.271 INFO:tasks.ceph:[global] ms tcp read timeout = 5 2024-09-16T03:38:19.271 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:38:19.271 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T03:38:19.271 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T03:38:19.272 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T03:38:19.272 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T03:38:19.272 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T03:38:19.272 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T03:38:19.272 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T03:38:19.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T03:38:19.326 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T03:38:19.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T03:38:19.376 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:38:19.402 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.76'), ('mon.b', '172.21.15.103'), ('mon.c', '172.21.15.119')] 2024-09-16T03:38:19.403 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.76,172.21.15.103,172.21.15.119', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-16T03:38:19.404 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:38:19.404 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:38:19.458 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.76 --add b 172.21.15.103 --add c 172.21.15.119 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:19.545 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: generated fsid ecf7a42a-e941-4b24-8693-0ff8b6eb41b1 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:setting min_mon_release = quincy 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 0 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:fsid ecf7a42a-e941-4b24-8693-0ff8b6eb41b1 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:last_changed 2024-09-16T03:38:19.544471+0000 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-09-16T03:38:19.544471+0000 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:min_mon_release 17 (quincy) 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:election_strategy: 3 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-09-16T03:38:19.546 INFO:teuthology.orchestra.run.smithi076.stdout:2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.c 2024-09-16T03:38:19.547 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T03:38:19.548 DEBUG:teuthology.orchestra.run.smithi076:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:38:19.563 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ecf7a42a-e941-4b24-8693-0ff8b6eb41b1... 2024-09-16T03:38:19.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:38:19.606 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:38:19.608 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:38:19.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:38:19.646 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2024-09-16T03:38:19.646 INFO:teuthology.orchestra.run.smithi076.stdout: chdir = "" 2024-09-16T03:38:19.646 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:38:19.646 INFO:teuthology.orchestra.run.smithi076.stdout: auth supported = cephx 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: filestore xattr use omap = true 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: mon clock drift allowed = 1.000 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: osd crush chooseleaf type = 0 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: auth debug = true 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on old message = true 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on bug = true 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:38:19.647 INFO:teuthology.orchestra.run.smithi076.stdout: mon pg warn max object skew = 0 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default size = 2 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow primary affinity = true 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow pg remap = true 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on no sortbitwise = false 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on too few osds = false 2024-09-16T03:38:19.648 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: mon_allow_pool_size_one = true 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: osd default data pool replay window = 5 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: mon allow pool delete = true 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: mon cluster log file level = debug 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: debug asserts on shutdown = true 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: mon health detail to clog = false 2024-09-16T03:38:19.649 INFO:teuthology.orchestra.run.smithi076.stdout: mon host = "172.21.15.76,172.21.15.103,172.21.15.119" 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: mon client directed command retry = 5 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: mon election default strategy = 3 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: ms inject socket failures = 2500 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: ms tcp read timeout = 5 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = ecf7a42a-e941-4b24-8693-0ff8b6eb41b1 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout:[osd] 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: osd journal size = 100 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub load threshold = 5.0 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub max interval = 120 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:38:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: osd recover clone overlap = true 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: osd recovery max chunk = 1048576 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug shutdown = true 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug op order = true 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify stray on activate = true 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug trim objects = true 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: osd open classes on start = true 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug pg log writeout = true 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.651 INFO:teuthology.orchestra.run.smithi076.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: osd map max advance = 10 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: journal zero on create = true 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: filestore apply finisher threads = 3 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: bdev debug aio = true 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug misdirected ops = true 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: debug monc = 20 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: bdev async discard = True 2024-09-16T03:38:19.652 INFO:teuthology.orchestra.run.smithi076.stdout: bdev enable discard = True 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore block size = 96636764160 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore fsck on mount = True 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore zero block detection = True 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluefs = 20 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluestore = 20 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: debug osd = 20 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: debug rocksdb = 10 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd full ratio = 0.9 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:38:19.653 INFO:teuthology.orchestra.run.smithi076.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug reject backfill probability = 0.1 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify cached snaps = True 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify missing on start = True 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd max backfills = 3 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd max markdown count = 1000 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd mclock override recovery settings = True 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd memory target = 1610612736 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd objectstore = bluestore 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd op queue = debug_random 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd op queue cut off = debug_random 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub min interval = 60 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: osd shutdown pgref assert = True 2024-09-16T03:38:19.654 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr] 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: debug mgr = 20 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: mgr/telemetry/nag = false 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout:[mon] 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2024-09-16T03:38:19.655 INFO:teuthology.orchestra.run.smithi076.stdout: debug paxos = 20 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: mon data avail warn = 5 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: mon mgr mkfs grace = 240 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd reporter subtree level = osd 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd prime pg temp = true 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:38:19.656 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: # 1m isn't quite enough 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: mon scrub interval = 300 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout:[client] 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: rgw cache enabled = true 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable ops log = true 2024-09-16T03:38:19.657 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable usage log = true 2024-09-16T03:38:19.658 INFO:teuthology.orchestra.run.smithi076.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:38:19.658 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:38:19.658 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.a] 2024-09-16T03:38:19.658 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.b] 2024-09-16T03:38:19.658 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.c] 2024-09-16T03:38:19.659 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-09-16T03:38:19.659 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-09-16T03:38:19.659 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:38:19.659 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-09-16T03:38:19.659 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.659 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-09-16T03:38:19.659 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.659 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-09-16T03:38:19.659 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-09-16T03:38:19.660 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.661 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = "172.21.15.76,172.21.15.103,172.21.15.119" 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: mon client directed command retry = 5 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: mon election default strategy = 3 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: ms inject socket failures = 2500 2024-09-16T03:38:19.662 INFO:teuthology.orchestra.run.smithi103.stdout: ms tcp read timeout = 5 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = ecf7a42a-e941-4b24-8693-0ff8b6eb41b1 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 120 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.663 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug trim objects = true 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-09-16T03:38:19.664 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: debug monc = 20 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: bdev async discard = True 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: bdev enable discard = True 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore zero block detection = True 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 20 2024-09-16T03:38:19.665 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 20 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 10 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug reject backfill probability = 0.1 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify cached snaps = True 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify missing on start = True 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: osd max backfills = 3 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: osd max markdown count = 1000 2024-09-16T03:38:19.666 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock override recovery settings = True 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: osd memory target = 1610612736 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: osd op queue = debug_random 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: osd op queue cut off = debug_random 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub min interval = 60 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: osd shutdown pgref assert = True 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-09-16T03:38:19.667 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-09-16T03:38:19.668 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.669 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout: mon scrub interval = 300 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.b] 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.c] 2024-09-16T03:38:19.670 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:38:19.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.76,172.21.15.103,172.21.15.119" 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: mon client directed command retry = 5 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: mon election default strategy = 3 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject socket failures = 2500 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: ms tcp read timeout = 5 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = ecf7a42a-e941-4b24-8693-0ff8b6eb41b1 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.674 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 120 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-09-16T03:38:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 20 2024-09-16T03:38:19.676 INFO:teuthology.orchestra.run.smithi121.stdout: bdev async discard = True 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: bdev enable discard = True 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore zero block detection = True 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 20 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 20 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 10 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:38:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug reject backfill probability = 0.1 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify cached snaps = True 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify missing on start = True 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd max backfills = 3 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd max markdown count = 1000 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock override recovery settings = True 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd memory target = 1610612736 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue = debug_random 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue cut off = debug_random 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub min interval = 60 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd shutdown pgref assert = True 2024-09-16T03:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-09-16T03:38:19.679 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:38:19.680 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub interval = 300 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:38:19.681 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-09-16T03:38:19.682 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.683 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:38:19.684 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "172.21.15.76,172.21.15.103,172.21.15.119" 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: mon client directed command retry = 5 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: mon election default strategy = 3 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: ms inject socket failures = 2500 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: ms tcp read timeout = 5 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:38:19.685 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = ecf7a42a-e941-4b24-8693-0ff8b6eb41b1 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 120 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug trim objects = true 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: debug monc = 20 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: bdev async discard = True 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: bdev enable discard = True 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore zero block detection = True 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 20 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 20 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-16T03:38:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 10 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug reject backfill probability = 0.1 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify cached snaps = True 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify missing on start = True 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: osd max backfills = 3 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: osd max markdown count = 1000 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock override recovery settings = True 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: osd memory target = 1610612736 2024-09-16T03:38:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: osd op queue = debug_random 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: osd op queue cut off = debug_random 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub min interval = 60 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: osd shutdown pgref assert = True 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:38:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:38:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: mon scrub interval = 300 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-16T03:38:19.692 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-09-16T03:38:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-09-16T03:38:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-09-16T03:38:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-09-16T03:38:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:38:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:38:19.693 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-09-16T03:38:19.693 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2024-09-16T03:38:19.693 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2024-09-16T03:38:19.695 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T03:38:19.696 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T03:38:19.790 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T03:38:19.790 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:38:19.791 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T03:38:19.806 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:38:19.806 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T03:38:19.862 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:38:19.862 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:38:19.862 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:38:19.862 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:38:19.937 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:38:19.937 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:19.993 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:38:19.993 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:38:19.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:38:19.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:38:20.034 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:38:20.034 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:20.091 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:38:20.091 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:38:20.092 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:38:20.092 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:38:20.127 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:38:20.127 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:20.182 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:38:20.182 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:38:20.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:38:20.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:38:20.220 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:38:20.220 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:20.277 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T03:38:20.277 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T03:38:20.277 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi076.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T03:38:20.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:38:20.396 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:38:20.399 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T03:38:20.399 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T03:38:20.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T03:38:20.440 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T03:38:20.453 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T03:38:20.453 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-16T03:38:20.453 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T03:38:20.479 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:20.479 INFO:tasks.ceph:role: osd.0 2024-09-16T03:38:20.479 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:38:20.479 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:38:20.551 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:20.552 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:20.552 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:20.552 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:20.552 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:20.552 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:20.552 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:20.552 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:20.552 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:20.552 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:21.057 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-09-16T03:38:21.060 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T03:38:21.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T03:38:21.211 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T03:38:21.245 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T03:38:21.312 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:21.312 INFO:tasks.ceph:role: osd.4 2024-09-16T03:38:21.312 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:38:21.312 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:38:21.385 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:21.385 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:21.385 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:21.385 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:21.385 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:21.385 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:21.385 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:21.385 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:21.385 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:21.385 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:21.830 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-09-16T03:38:21.832 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T03:38:21.832 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T03:38:21.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T03:38:21.948 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-16T03:38:22.013 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:22.014 INFO:tasks.ceph:role: osd.8 2024-09-16T03:38:22.014 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:38:22.014 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:38:22.085 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:22.085 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:22.085 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:22.085 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:22.086 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:22.086 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:22.086 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:22.086 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:22.086 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:22.086 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:22.585 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-09-16T03:38:22.587 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T03:38:22.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-16T03:38:22.710 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-16T03:38:22.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-16T03:38:22.806 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:22.806 INFO:tasks.ceph:role: osd.12 2024-09-16T03:38:22.806 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:38:22.806 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:38:22.878 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:22.878 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:22.879 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:22.879 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:22.879 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:22.879 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:22.879 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:22.879 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:22.879 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:22.879 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:23.361 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-09-16T03:38:23.363 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T03:38:23.364 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-16T03:38:23.475 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-16T03:38:23.505 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:23.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:23.607+0000 7f77bd00f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T03:38:23.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:23.613+0000 7f77bd00f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T03:38:23.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:23.613+0000 7f77bd00f640 -1 bdev(0x55a3691b9800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T03:38:23.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:23.613+0000 7f77bd00f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T03:38:27.265 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T03:38:27.296 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:27.388+0000 7f7d70e39640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:27.388+0000 7f7d70e39640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:27.388+0000 7f7d70e39640 -1 bdev(0x563ee27cb800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:27.388+0000 7f7d70e39640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T03:38:31.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T03:38:31.319 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:31.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:31.408+0000 7f7493958640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-16T03:38:31.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:31.408+0000 7f7493958640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-16T03:38:31.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:31.408+0000 7f7493958640 -1 bdev(0x55d0a44c9800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-16T03:38:31.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:31.408+0000 7f7493958640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-16T03:38:35.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-16T03:38:35.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:35.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:35.181+0000 7f587261a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-16T03:38:35.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:35.181+0000 7f587261a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-16T03:38:35.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:35.181+0000 7f587261a640 -1 bdev(0x557cde3ff800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-16T03:38:35.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:38:35.181+0000 7f587261a640 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-16T03:38:38.584 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-16T03:38:38.614 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T03:38:38.644 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:38.644 INFO:tasks.ceph:role: osd.1 2024-09-16T03:38:38.644 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:38:38.644 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:38:38.721 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:38.721 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:38.721 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:38.721 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:38.721 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:38.721 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:38.721 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:38.721 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:38.721 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:38.721 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:39.230 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-16T03:38:39.233 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-16T03:38:39.233 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-16T03:38:39.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T03:38:39.402 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T03:38:39.469 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:39.469 INFO:tasks.ceph:role: osd.5 2024-09-16T03:38:39.469 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:38:39.469 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:38:39.543 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:39.543 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:39.543 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:39.543 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:39.543 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:39.543 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:39.543 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:39.543 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:39.543 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:39.543 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:40.002 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-16T03:38:40.004 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-16T03:38:40.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T03:38:40.085 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T03:38:40.154 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-16T03:38:40.219 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:40.219 INFO:tasks.ceph:role: osd.9 2024-09-16T03:38:40.219 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:38:40.219 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:38:40.291 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:40.292 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:40.292 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:40.292 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:40.292 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:40.292 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:40.292 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:40.292 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:40.292 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:40.292 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:40.780 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-16T03:38:40.782 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-16T03:38:40.783 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-16T03:38:40.913 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-16T03:38:40.943 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-16T03:38:41.008 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:41.008 INFO:tasks.ceph:role: osd.13 2024-09-16T03:38:41.008 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:38:41.008 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:38:41.081 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:41.081 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:41.081 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:41.081 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:41.081 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:41.081 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:41.081 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:41.081 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:41.081 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:41.082 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:41.599 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-16T03:38:41.601 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-16T03:38:41.601 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-16T03:38:41.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-16T03:38:41.763 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:41.866+0000 7ff5b25d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:41.866+0000 7ff5b25d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:41.867+0000 7ff5b25d9640 -1 bdev(0x562af84d7800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:41.867+0000 7ff5b25d9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T03:38:45.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T03:38:45.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:45.650+0000 7f7fd1244640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:45.650+0000 7f7fd1244640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:45.651+0000 7f7fd1244640 -1 bdev(0x564926451800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:45.651+0000 7f7fd1244640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T03:38:49.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T03:38:49.578 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:49.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:49.664+0000 7fb1a87c5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-16T03:38:49.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:49.665+0000 7fb1a87c5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-16T03:38:49.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:49.665+0000 7fb1a87c5640 -1 bdev(0x55bdcaeed800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-16T03:38:49.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:49.665+0000 7fb1a87c5640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-16T03:38:53.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-16T03:38:53.587 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:38:53.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:53.676+0000 7f282a0b0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-16T03:38:53.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:53.676+0000 7f282a0b0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-16T03:38:53.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:53.676+0000 7f282a0b0640 -1 bdev(0x55b692bb5800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-16T03:38:53.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T03:38:53.676+0000 7f282a0b0640 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-16T03:38:57.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-16T03:38:57.602 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T03:38:57.636 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:57.636 INFO:tasks.ceph:role: osd.2 2024-09-16T03:38:57.637 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:38:57.639 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:38:57.710 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:57.710 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:57.710 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:57.710 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:57.710 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:57.710 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:57.710 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:57.711 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:57.711 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:57.711 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:58.207 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-09-16T03:38:58.209 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-16T03:38:58.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-16T03:38:58.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T03:38:58.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T03:38:58.461 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:58.461 INFO:tasks.ceph:role: osd.6 2024-09-16T03:38:58.461 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:38:58.462 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:38:58.536 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:58.536 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:58.536 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:58.536 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:58.537 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:58.537 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:58.537 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:58.537 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:58.537 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:58.537 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:58.990 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-09-16T03:38:58.992 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-16T03:38:58.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-16T03:38:59.081 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T03:38:59.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-16T03:38:59.179 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:59.180 INFO:tasks.ceph:role: osd.10 2024-09-16T03:38:59.180 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:38:59.180 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:38:59.253 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:38:59.253 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:38:59.254 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:38:59.254 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:38:59.254 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:38:59.254 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:38:59.254 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:38:59.254 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:38:59.254 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:38:59.254 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:38:59.749 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-09-16T03:38:59.751 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-16T03:38:59.751 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-16T03:38:59.870 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-16T03:38:59.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-16T03:38:59.969 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:38:59.969 INFO:tasks.ceph:role: osd.14 2024-09-16T03:38:59.969 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:38:59.969 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:39:00.043 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:39:00.044 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:39:00.044 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:39:00.044 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:39:00.044 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:39:00.044 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:39:00.044 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:39:00.044 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:39:00.044 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:39:00.044 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:39:00.558 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-09-16T03:39:00.560 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-16T03:39:00.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-16T03:39:00.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-16T03:39:00.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:00.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:00.814+0000 7f8de9844640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T03:39:00.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:00.814+0000 7f8de9844640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T03:39:00.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:00.814+0000 7f8de9844640 -1 bdev(0x56390680b800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T03:39:00.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:00.814+0000 7f8de9844640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T03:39:04.467 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T03:39:04.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:04.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:04.584+0000 7f57f8415640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T03:39:04.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:04.585+0000 7f57f8415640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T03:39:04.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:04.585+0000 7f57f8415640 -1 bdev(0x55dcca6cf800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-16T03:39:04.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:04.585+0000 7f57f8415640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T03:39:08.005 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T03:39:08.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:08.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:08.122+0000 7f57de5dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-16T03:39:08.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:08.122+0000 7f57de5dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-16T03:39:08.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:08.122+0000 7f57de5dd640 -1 bdev(0x557c7f689800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-16T03:39:08.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:08.122+0000 7f57de5dd640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-16T03:39:12.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-16T03:39:12.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:12.148+0000 7faf0073e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:12.148+0000 7faf0073e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:12.148+0000 7faf0073e640 -1 bdev(0x559c484e9800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-16T03:39:12.149+0000 7faf0073e640 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-16T03:39:15.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-16T03:39:15.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T03:39:15.880 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:39:15.880 INFO:tasks.ceph:role: osd.3 2024-09-16T03:39:15.880 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:39:15.880 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:39:15.956 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:39:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:39:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:39:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:39:15.957 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:39:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:39:15.957 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:39:15.957 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:39:15.957 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:39:15.957 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-09-16T03:39:16.479 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-09-16T03:39:16.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T03:39:16.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T03:39:16.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T03:39:16.686 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:39:16.686 INFO:tasks.ceph:role: osd.7 2024-09-16T03:39:16.686 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:39:16.686 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:39:17.223 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-09-16T03:39:17.225 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-09-16T03:39:17.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-16T03:39:17.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T03:39:17.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-16T03:39:17.405 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:39:17.405 INFO:tasks.ceph:role: osd.11 2024-09-16T03:39:17.405 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:39:17.406 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:39:17.480 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:39:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:39:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:39:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:39:17.480 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:39:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:39:17.480 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:39:17.480 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:39:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:39:17.481 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:39:17.971 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-09-16T03:39:17.973 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-09-16T03:39:17.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-16T03:39:18.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-16T03:39:18.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-16T03:39:18.215 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:39:18.215 INFO:tasks.ceph:role: osd.15 2024-09-16T03:39:18.215 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:39:18.216 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:39:18.287 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:39:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:39:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:39:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:39:18.287 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:39:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:39:18.287 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:39:18.288 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:39:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:39:18.288 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:39:18.812 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-09-16T03:39:18.814 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-09-16T03:39:18.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-16T03:39:18.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-16T03:39:18.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:19.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:19.084+0000 7f458abfd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T03:39:19.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:19.085+0000 7f458abfd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T03:39:19.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:19.085+0000 7f458abfd640 -1 bdev(0x5642ba211800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T03:39:19.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:19.085+0000 7f458abfd640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T03:39:22.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T03:39:23.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:23.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:23.111+0000 7fc17f019640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T03:39:23.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:23.111+0000 7fc17f019640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T03:39:23.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:23.111+0000 7fc17f019640 -1 bdev(0x5582c028f800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-16T03:39:23.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:23.111+0000 7fc17f019640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T03:39:26.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T03:39:26.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:26.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:26.886+0000 7f6ef29c3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-16T03:39:26.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:26.887+0000 7f6ef29c3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-16T03:39:26.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:26.887+0000 7f6ef29c3640 -1 bdev(0x559dd5f03800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-16T03:39:26.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:26.887+0000 7f6ef29c3640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-16T03:39:30.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-16T03:39:30.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:30.168+0000 7f45af422640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-16T03:39:30.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:30.168+0000 7f45af422640 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-16T03:39:30.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:30.169+0000 7f45af422640 -1 bdev(0x55821d9d1800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-16T03:39:30.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-16T03:39:30.169+0000 7f45af422640 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-16T03:39:33.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-16T03:39:33.618 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T03:39:33.619 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:39:33.619 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T03:39:33.648 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:39:33.648 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T03:39:33.717 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:39:33.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T03:39:33.784 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:39:33.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-16T03:39:33.851 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:39:33.851 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-16T03:39:33.917 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:39:33.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T03:39:33.948 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:39:33.948 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T03:39:34.017 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:39:34.017 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-16T03:39:34.087 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:39:34.087 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-16T03:39:34.154 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:39:34.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T03:39:34.184 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:39:34.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T03:39:34.254 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:39:34.254 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-16T03:39:34.321 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:39:34.321 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-16T03:39:34.387 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:39:34.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T03:39:34.418 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:39:34.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T03:39:34.487 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:39:34.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T03:39:34.554 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:39:34.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-16T03:39:34.620 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:39:34.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-16T03:39:34.687 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:39:34.687 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T03:39:34.742 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T03:39:34.743 DEBUG:teuthology.orchestra.run.smithi076:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:39:34.745 DEBUG:teuthology.orchestra.run.smithi103:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:39:34.748 DEBUG:teuthology.orchestra.run.smithi119:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:39:34.770 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr.y] 2024-09-16T03:39:34.770 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAsqOdmbc2uExAAaR5b8n8X0CtmjKg73adzmg== 2024-09-16T03:39:34.770 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.0] 2024-09-16T03:39:34.770 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAvqOdm9WxEJBAA3OTVwABLdDR1NanvHj7VCg== 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.4] 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAzqOdm6QYsFxAA7hJoTIOx4EHUI5KPq3upYQ== 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.8] 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA3qOdm5thgGBAABMfW4rtzJAZ7LjKHALtQ4g== 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.12] 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA7qOdmOZTUChAAbER0DBGWnalFn4jYtN6Dyg== 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.1] 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBBqOdmWCuxMxAAn3COx/XTXVUHZ+25bk1PRA== 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.5] 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBFqOdmNI3RJhAACl4jaruvwO8p6/oWw6j/Og== 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.9] 2024-09-16T03:39:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBJqOdmsteqJxAArDUNhwV5LWOASYAaxG5ClA== 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.13] 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBNqOdm1+5TKBAAKz8Ih2Fv8Shn0wHtuCIpOA== 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.2] 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBUqOdmfEKZMBAAlYZCkMnsYHUo/L0LO9xTTA== 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.6] 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBYqOdmrWnpIhAA0PZQGMHOBcOhvgi4LMNeWQ== 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.10] 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBcqOdmu3paBxAAe950i+Ute6p+RaNZLU6uGg== 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.14] 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBgqOdmukjqCBAAeXS70kMBqEW9UUC8k+5HCw== 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr.x] 2024-09-16T03:39:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAsqOdmyHeZFxAA6nCcuaP4UsqlpRYa3oGUXg== 2024-09-16T03:39:34.773 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.3] 2024-09-16T03:39:34.773 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBnqOdmRDcqBRAA48Trd3LCK/B2nma825SGyQ== 2024-09-16T03:39:34.773 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.7] 2024-09-16T03:39:34.773 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBrqOdm6+G5BhAAxqm8QRHPUPPsk5grGY1/zw== 2024-09-16T03:39:34.773 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.11] 2024-09-16T03:39:34.773 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBuqOdm4MT0NBAAnEonjrCOygUwskJqOeG1OA== 2024-09-16T03:39:34.773 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.15] 2024-09-16T03:39:34.773 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQByqOdmVTMaChAAM1NULlKMOYpypwueh893NA== 2024-09-16T03:39:34.773 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2024-09-16T03:39:34.773 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAsqOdmTrQzGhAA5djFAMDYIvL3SB0/3xSvEg== 2024-09-16T03:39:34.774 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.y] 2024-09-16T03:39:34.774 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAsqOdmbc2uExAAaR5b8n8X0CtmjKg73adzmg== 2024-09-16T03:39:34.774 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2024-09-16T03:39:34.774 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAvqOdm9WxEJBAA3OTVwABLdDR1NanvHj7VCg== 2024-09-16T03:39:34.774 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.4] 2024-09-16T03:39:34.774 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAzqOdm6QYsFxAA7hJoTIOx4EHUI5KPq3upYQ== 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.8] 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA3qOdm5thgGBAABMfW4rtzJAZ7LjKHALtQ4g== 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.12] 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA7qOdmOZTUChAAbER0DBGWnalFn4jYtN6Dyg== 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBBqOdmWCuxMxAAn3COx/XTXVUHZ+25bk1PRA== 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.5] 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBFqOdmNI3RJhAACl4jaruvwO8p6/oWw6j/Og== 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.9] 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBJqOdmsteqJxAArDUNhwV5LWOASYAaxG5ClA== 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.13] 2024-09-16T03:39:34.775 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBNqOdm1+5TKBAAKz8Ih2Fv8Shn0wHtuCIpOA== 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBUqOdmfEKZMBAAlYZCkMnsYHUo/L0LO9xTTA== 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.6] 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBYqOdmrWnpIhAA0PZQGMHOBcOhvgi4LMNeWQ== 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.10] 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBcqOdmu3paBxAAe950i+Ute6p+RaNZLU6uGg== 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.14] 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBgqOdmukjqCBAAeXS70kMBqEW9UUC8k+5HCw== 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.x] 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAsqOdmyHeZFxAA6nCcuaP4UsqlpRYa3oGUXg== 2024-09-16T03:39:34.776 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.3] 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBnqOdmRDcqBRAA48Trd3LCK/B2nma825SGyQ== 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.7] 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBrqOdm6+G5BhAAxqm8QRHPUPPsk5grGY1/zw== 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.11] 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBuqOdm4MT0NBAAnEonjrCOygUwskJqOeG1OA== 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.15] 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQByqOdmVTMaChAAM1NULlKMOYpypwueh893NA== 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAsqOdmTrQzGhAA5djFAMDYIvL3SB0/3xSvEg== 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.y] 2024-09-16T03:39:34.777 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAsqOdmbc2uExAAaR5b8n8X0CtmjKg73adzmg== 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAvqOdm9WxEJBAA3OTVwABLdDR1NanvHj7VCg== 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.4] 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAzqOdm6QYsFxAA7hJoTIOx4EHUI5KPq3upYQ== 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.8] 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQA3qOdm5thgGBAABMfW4rtzJAZ7LjKHALtQ4g== 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.12] 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQA7qOdmOZTUChAAbER0DBGWnalFn4jYtN6Dyg== 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBBqOdmWCuxMxAAn3COx/XTXVUHZ+25bk1PRA== 2024-09-16T03:39:34.778 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.5] 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBFqOdmNI3RJhAACl4jaruvwO8p6/oWw6j/Og== 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.9] 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBJqOdmsteqJxAArDUNhwV5LWOASYAaxG5ClA== 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.13] 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBNqOdm1+5TKBAAKz8Ih2Fv8Shn0wHtuCIpOA== 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBUqOdmfEKZMBAAlYZCkMnsYHUo/L0LO9xTTA== 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.6] 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBYqOdmrWnpIhAA0PZQGMHOBcOhvgi4LMNeWQ== 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.10] 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBcqOdmu3paBxAAe950i+Ute6p+RaNZLU6uGg== 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.14] 2024-09-16T03:39:34.779 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBgqOdmukjqCBAAeXS70kMBqEW9UUC8k+5HCw== 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.x] 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAsqOdmyHeZFxAA6nCcuaP4UsqlpRYa3oGUXg== 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.3] 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBnqOdmRDcqBRAA48Trd3LCK/B2nma825SGyQ== 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.7] 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBrqOdm6+G5BhAAxqm8QRHPUPPsk5grGY1/zw== 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.11] 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBuqOdm4MT0NBAAnEonjrCOygUwskJqOeG1OA== 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.15] 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQByqOdmVTMaChAAM1NULlKMOYpypwueh893NA== 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-09-16T03:39:34.780 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAsqOdmTrQzGhAA5djFAMDYIvL3SB0/3xSvEg== 2024-09-16T03:39:34.782 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:39:34.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:39:34.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:39:34.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:34.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:34.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:34.940 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:34.943 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:34.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:34.996 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:34.999 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.001 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.051 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.054 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.106 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.108 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.111 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.161 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.163 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.216 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.218 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.271 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.276 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.326 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.329 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.331 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.382 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.384 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.439 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.441 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.493 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.546 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:39:35.548 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:39:35.550 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:39:35.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.603 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.658 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.660 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.711 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.713 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.766 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:39:35.821 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:39:35.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:39:35.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:39:35.879 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T03:39:35.880 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T03:39:35.905 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:39:36.003 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T03:39:36.027 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T03:39:36.053 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:39:36.149 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T03:39:36.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T03:39:36.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:39:36.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T03:39:36.319 DEBUG:teuthology.orchestra.run.smithi076:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:36.322 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:36.324 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:39:36.374 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T03:39:36.375 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:39:36.375 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:39:36.377 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:39:36.378 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:39:36.378 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:39:36.380 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:39:36.380 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:39:36.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:39:36.417 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:39:36.417 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T03:39:36.417 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T03:39:36.417 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T03:39:36.419 INFO:tasks.ceph.mgr.y:Started 2024-09-16T03:39:36.420 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T03:39:36.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T03:39:36.422 INFO:tasks.ceph.mgr.x:Started 2024-09-16T03:39:36.422 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T03:39:36.423 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T03:39:42.783 INFO:teuthology.orchestra.run.smithi076.stderr:adjusted tunables profile to default 2024-09-16T03:39:42.795 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T03:39:42.795 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T03:39:42.795 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:39:42.796 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T03:39:42.821 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:39:42.821 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T03:39:42.887 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:39:42.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-16T03:39:42.950 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:42.948+0000 7f9de8010040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T03:39:42.952 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T03:39:42.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-16T03:39:42.986 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:42.983+0000 7fc76fe99040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T03:39:42.992 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:42.991+0000 7f9de8010040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T03:39:43.010 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:39:43.010 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T03:39:43.029 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:43.026+0000 7fc76fe99040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T03:39:43.041 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:39:43.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T03:39:43.108 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:39:43.108 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-16T03:39:43.141 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:43.136+0000 7f9de8010040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T03:39:43.169 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:43.166+0000 7fc76fe99040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T03:39:43.176 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T03:39:43.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-16T03:39:43.188 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:43.186+0000 7f9de8010040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T03:39:43.219 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:43.216+0000 7fc76fe99040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T03:39:43.243 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:39:43.243 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T03:39:43.273 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:39:43.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T03:39:43.338 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:43.336+0000 7f9de8010040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T03:39:43.343 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:39:43.343 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-16T03:39:43.380 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:43.377+0000 7fc76fe99040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T03:39:43.411 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-16T03:39:43.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-16T03:39:43.448 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:43.446+0000 7f9de8010040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T03:39:43.477 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:39:43.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T03:39:43.489 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:43.487+0000 7fc76fe99040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T03:39:43.502 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:39:43.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T03:39:43.567 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:39:43.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-16T03:39:43.632 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-16T03:39:43.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-16T03:39:43.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 8820e1f9-b619-44a5-abb1-7cfa97fcef00 0 2024-09-16T03:39:43.783 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:43.781+0000 7f9de8010040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T03:39:43.816 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:43.814+0000 7fc76fe99040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T03:39:43.836 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:43.834+0000 7f9de8010040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T03:39:43.866 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:43.863+0000 7fc76fe99040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T03:39:43.916 INFO:teuthology.orchestra.run.smithi121.stdout:0 2024-09-16T03:39:43.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 0f58ae45-e2ff-4c4f-9875-7d34ec312cbe 1 2024-09-16T03:39:43.993 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:43.991+0000 7f9de8010040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T03:39:44.015 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:44.013+0000 7fc76fe99040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T03:39:44.108 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-09-16T03:39:44.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new b369fe92-86b8-45b3-a418-024f0e4becd7 2 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi121.stdout:2 2024-09-16T03:39:44.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new b9521bf1-46ef-402c-8472-c1b476280b23 3 2024-09-16T03:39:44.485 INFO:teuthology.orchestra.run.smithi121.stdout:3 2024-09-16T03:39:44.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 15bec6a1-fc13-4822-8cfa-899b83886c10 4 2024-09-16T03:39:44.674 INFO:teuthology.orchestra.run.smithi121.stdout:4 2024-09-16T03:39:44.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new e9b234ec-bfa4-4f72-adab-295e067b2e61 5 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi121.stdout:5 2024-09-16T03:39:44.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 6d4ae9fb-8d8f-451d-a12f-1265302b03ab 6 2024-09-16T03:39:44.974 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:44.971+0000 7fc76fe99040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T03:39:44.986 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:44.985+0000 7f9de8010040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T03:39:45.027 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:45.024+0000 7fc76fe99040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T03:39:45.040 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:45.039+0000 7f9de8010040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T03:39:45.049 INFO:teuthology.orchestra.run.smithi121.stdout:6 2024-09-16T03:39:45.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 2c9456ed-00ec-46aa-814e-f85eb441fc4f 7 2024-09-16T03:39:45.240 INFO:teuthology.orchestra.run.smithi121.stdout:7 2024-09-16T03:39:45.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new b618a02a-8628-48f5-bb5f-f7591aafa118 8 2024-09-16T03:39:45.393 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:45.390+0000 7fc76fe99040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T03:39:45.425 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:45.424+0000 7f9de8010040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi121.stdout:8 2024-09-16T03:39:45.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new dc3e5be7-f012-47e6-ba35-0e3354ca0c18 9 2024-09-16T03:39:45.478 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:45.475+0000 7fc76fe99040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T03:39:45.517 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:45.515+0000 7f9de8010040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T03:39:45.584 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:45.581+0000 7fc76fe99040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T03:39:45.615 INFO:teuthology.orchestra.run.smithi121.stdout:9 2024-09-16T03:39:45.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 4e70a7a8-7ba8-42ab-a45a-e0564b561619 10 2024-09-16T03:39:45.629 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:45.627+0000 7f9de8010040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T03:39:45.630 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:45.627+0000 7fc76fe99040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T03:39:45.677 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:45.676+0000 7f9de8010040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T03:39:45.745 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:45.742+0000 7fc76fe99040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T03:39:45.795 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:45.793+0000 7f9de8010040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T03:39:45.798 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:45.795+0000 7fc76fe99040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi121.stdout:10 2024-09-16T03:39:45.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new d16bc493-335f-41c2-bd68-cbad93c49a5d 11 2024-09-16T03:39:45.843 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:45.841+0000 7fc76fe99040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T03:39:45.858 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:45.857+0000 7f9de8010040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T03:39:45.905 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:45.904+0000 7f9de8010040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T03:39:45.956 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:45.953+0000 7fc76fe99040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T03:39:45.990 INFO:teuthology.orchestra.run.smithi121.stdout:11 2024-09-16T03:39:46.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1 12 2024-09-16T03:39:46.006 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:46.003+0000 7fc76fe99040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T03:39:46.024 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:46.023+0000 7f9de8010040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T03:39:46.057 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:46.054+0000 7fc76fe99040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T03:39:46.077 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:46.075+0000 7f9de8010040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T03:39:46.125 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:46.123+0000 7fc76fe99040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T03:39:46.131 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:46.129+0000 7f9de8010040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T03:39:46.169 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:46.167+0000 7fc76fe99040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T03:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout:12 2024-09-16T03:39:46.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 61e9204b-8269-4c35-b409-5cfbd5e6abbb 13 2024-09-16T03:39:46.201 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:46.199+0000 7f9de8010040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T03:39:46.214 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:46.211+0000 7fc76fe99040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T03:39:46.250 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:46.248+0000 7f9de8010040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T03:39:46.296 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:46.294+0000 7f9de8010040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T03:39:46.378 INFO:teuthology.orchestra.run.smithi121.stdout:13 2024-09-16T03:39:46.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 0bd5338e-eb99-44cc-9f5f-1c8ef903d02c 14 2024-09-16T03:39:46.565 INFO:teuthology.orchestra.run.smithi121.stdout:14 2024-09-16T03:39:46.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new f240405e-d268-41cc-bd99-5a2ca71a85a9 15 2024-09-16T03:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout:15 2024-09-16T03:39:46.763 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T03:39:46.763 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T03:39:46.765 INFO:tasks.ceph.osd.0:Started 2024-09-16T03:39:46.765 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T03:39:46.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T03:39:46.767 INFO:tasks.ceph.osd.4:Started 2024-09-16T03:39:46.768 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-16T03:39:46.768 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-16T03:39:46.769 INFO:tasks.ceph.osd.8:Started 2024-09-16T03:39:46.770 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-16T03:39:46.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-16T03:39:46.772 INFO:tasks.ceph.osd.12:Started 2024-09-16T03:39:46.772 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T03:39:46.772 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T03:39:46.775 INFO:tasks.ceph.osd.1:Started 2024-09-16T03:39:46.775 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T03:39:46.775 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T03:39:46.777 INFO:tasks.ceph.osd.5:Started 2024-09-16T03:39:46.777 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-16T03:39:46.777 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-16T03:39:46.779 INFO:tasks.ceph.osd.9:Started 2024-09-16T03:39:46.779 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-16T03:39:46.779 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-16T03:39:46.781 INFO:tasks.ceph.osd.13:Started 2024-09-16T03:39:46.781 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T03:39:46.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T03:39:46.784 INFO:tasks.ceph.osd.2:Started 2024-09-16T03:39:46.784 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T03:39:46.784 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T03:39:46.786 INFO:tasks.ceph.osd.6:Started 2024-09-16T03:39:46.787 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-16T03:39:46.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-16T03:39:46.789 INFO:tasks.ceph.osd.10:Started 2024-09-16T03:39:46.789 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-16T03:39:46.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-16T03:39:46.791 INFO:tasks.ceph.osd.14:Started 2024-09-16T03:39:46.792 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T03:39:46.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T03:39:46.795 INFO:tasks.ceph.osd.3:Started 2024-09-16T03:39:46.795 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T03:39:46.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T03:39:46.797 INFO:tasks.ceph.osd.7:Started 2024-09-16T03:39:46.797 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-16T03:39:46.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-16T03:39:46.799 INFO:tasks.ceph.osd.11:Started 2024-09-16T03:39:46.800 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-16T03:39:46.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-16T03:39:46.802 INFO:tasks.ceph.osd.15:Started 2024-09-16T03:39:46.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:39:46.908 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:46.905+0000 7fc76fe99040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T03:39:46.989 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:39:46.989 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":18,"fsid":"ecf7a42a-e941-4b24-8693-0ff8b6eb41b1","created":"2024-09-16T03:39:42.686123+0000","modified":"2024-09-16T03:39:46.749797+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:39:46.749797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8820e1f9-b619-44a5-abb1-7cfa97fcef00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0f58ae45-e2ff-4c4f-9875-7d34ec312cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b369fe92-86b8-45b3-a418-024f0e4becd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b9521bf1-46ef-402c-8472-c1b476280b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"15bec6a1-fc13-4822-8cfa-899b83886c10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e9b234ec-bfa4-4f72-adab-295e067b2e61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6d4ae9fb-8d8f-451d-a12f-1265302b03ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2c9456ed-00ec-46aa-814e-f85eb441fc4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"b618a02a-8628-48f5-bb5f-f7591aafa118","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"dc3e5be7-f012-47e6-ba35-0e3354ca0c18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"4e70a7a8-7ba8-42ab-a45a-e0564b561619","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d16bc493-335f-41c2-bd68-cbad93c49a5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"61e9204b-8269-4c35-b409-5cfbd5e6abbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"0bd5338e-eb99-44cc-9f5f-1c8ef903d02c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"f240405e-d268-41cc-bd99-5a2ca71a85a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:39:47.000 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T03:39:47.000 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T03:39:47.001 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-09-16T03:39:46.998+0000 7f9de8010040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T03:39:47.924 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:47.921+0000 7fc710767640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T03:39:47.924 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T03:39:47.921+0000 7fc710767640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T03:39:48.266 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:39:48.264+0000 7eff565b2640 -1 Falling back to public interface 2024-09-16T03:39:48.463 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:39:48.460+0000 7ff837702640 -1 Falling back to public interface 2024-09-16T03:39:48.477 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:39:48.475+0000 7f8f99811640 -1 Falling back to public interface 2024-09-16T03:39:48.481 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:39:48.479+0000 7fd24c727640 -1 Falling back to public interface 2024-09-16T03:39:48.489 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:39:48.487+0000 7f8ee1212640 -1 Falling back to public interface 2024-09-16T03:39:48.498 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:39:48.496+0000 7f86b3460640 -1 Falling back to public interface 2024-09-16T03:39:48.509 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:39:48.507+0000 7fc63c31d640 -1 Falling back to public interface 2024-09-16T03:39:48.612 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:39:48.696 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:39:48.693+0000 7f7e03ae2640 -1 Falling back to public interface 2024-09-16T03:39:48.707 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:39:48.705+0000 7fc58aa2c640 -1 Falling back to public interface 2024-09-16T03:39:48.733 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:39:48.730+0000 7fdb76485640 -1 Falling back to public interface 2024-09-16T03:39:48.736 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:39:48.734+0000 7f237fef1640 -1 Falling back to public interface 2024-09-16T03:39:48.746 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:39:48.743+0000 7fa9bfa0e640 -1 Falling back to public interface 2024-09-16T03:39:48.751 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:39:48.749+0000 7f100c60d640 -1 Falling back to public interface 2024-09-16T03:39:48.751 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:39:48.749+0000 7f92a0215640 -1 Falling back to public interface 2024-09-16T03:39:48.761 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:39:48.759+0000 7fe426614640 -1 Falling back to public interface 2024-09-16T03:39:48.761 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:39:48.759+0000 7fb17780f640 -1 Falling back to public interface 2024-09-16T03:39:48.892 INFO:teuthology.misc.health.smithi076.stdout: 2024-09-16T03:39:48.892 INFO:teuthology.misc.health.smithi076.stdout:{"epoch":18,"fsid":"ecf7a42a-e941-4b24-8693-0ff8b6eb41b1","created":"2024-09-16T03:39:42.686123+0000","modified":"2024-09-16T03:39:46.749797+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:39:46.749797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8820e1f9-b619-44a5-abb1-7cfa97fcef00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0f58ae45-e2ff-4c4f-9875-7d34ec312cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b369fe92-86b8-45b3-a418-024f0e4becd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b9521bf1-46ef-402c-8472-c1b476280b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"15bec6a1-fc13-4822-8cfa-899b83886c10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e9b234ec-bfa4-4f72-adab-295e067b2e61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6d4ae9fb-8d8f-451d-a12f-1265302b03ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2c9456ed-00ec-46aa-814e-f85eb441fc4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"b618a02a-8628-48f5-bb5f-f7591aafa118","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"dc3e5be7-f012-47e6-ba35-0e3354ca0c18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"4e70a7a8-7ba8-42ab-a45a-e0564b561619","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d16bc493-335f-41c2-bd68-cbad93c49a5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"61e9204b-8269-4c35-b409-5cfbd5e6abbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"0bd5338e-eb99-44cc-9f5f-1c8ef903d02c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"f240405e-d268-41cc-bd99-5a2ca71a85a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:39:48.904 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-16T03:39:52.070 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:39:52.068+0000 7eff565b2640 -1 osd.14 0 log_to_monitors true 2024-09-16T03:39:52.547 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:39:52.544+0000 7fd24c727640 -1 osd.1 0 log_to_monitors true 2024-09-16T03:39:52.755 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:39:52.752+0000 7ff837702640 -1 osd.4 0 log_to_monitors true 2024-09-16T03:39:52.910 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:39:52.908+0000 7f7e03ae2640 -1 osd.0 0 log_to_monitors true 2024-09-16T03:39:52.951 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:39:52.948+0000 7fc58aa2c640 -1 osd.13 0 log_to_monitors true 2024-09-16T03:39:53.006 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:39:53.003+0000 7f8ee1212640 -1 osd.9 0 log_to_monitors true 2024-09-16T03:39:53.011 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:39:53.009+0000 7f86b3460640 -1 osd.10 0 log_to_monitors true 2024-09-16T03:39:53.059 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:39:53.057+0000 7f8f99811640 -1 osd.5 0 log_to_monitors true 2024-09-16T03:39:53.061 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:39:53.058+0000 7fa9bfa0e640 -1 osd.12 0 log_to_monitors true 2024-09-16T03:39:53.065 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:39:53.064+0000 7fb17780f640 -1 osd.6 0 log_to_monitors true 2024-09-16T03:39:53.074 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:39:53.073+0000 7fc63c31d640 -1 osd.15 0 log_to_monitors true 2024-09-16T03:39:53.083 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:39:53.081+0000 7fe426614640 -1 osd.11 0 log_to_monitors true 2024-09-16T03:39:53.139 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:39:53.137+0000 7eff51d1e640 -1 osd.14 0 waiting for initial osdmap 2024-09-16T03:39:53.153 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:39:53.151+0000 7eff4e358640 -1 osd.14 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:53.232 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:39:53.230+0000 7f237fef1640 -1 osd.2 0 log_to_monitors true 2024-09-16T03:39:53.250 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:39:53.248+0000 7f92a0215640 -1 osd.3 0 log_to_monitors true 2024-09-16T03:39:53.260 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:39:53.258+0000 7fdb76485640 -1 osd.8 0 log_to_monitors true 2024-09-16T03:39:53.292 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:39:53.290+0000 7f100c60d640 -1 osd.7 0 log_to_monitors true 2024-09-16T03:39:54.150 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:39:54.148+0000 7fd247e93640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T03:39:54.255 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:39:54.253+0000 7f86aebcc640 -1 osd.10 0 waiting for initial osdmap 2024-09-16T03:39:54.286 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:39:54.284+0000 7f86ab206640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:54.400 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:39:54.397+0000 7fd2444cd640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:54.957 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:39:54.955+0000 7fb1729d6640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T03:39:54.972 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:39:54.970+0000 7fb16b008640 -1 osd.6 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.207 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:39:55.205+0000 7f10076ca640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T03:39:55.208 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:39:55.205+0000 7f237b65d640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T03:39:55.218 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:39:55.217+0000 7f2377c97640 -1 osd.2 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.219 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:39:55.218+0000 7fdb71bf1640 -1 osd.8 0 waiting for initial osdmap 2024-09-16T03:39:55.220 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:39:55.219+0000 7f1003d04640 -1 osd.7 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.241 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:39:55.240+0000 7fdb6e22b640 -1 osd.8 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.426 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:39:55.425+0000 7fe4226b3640 -1 osd.11 0 waiting for initial osdmap 2024-09-16T03:39:55.437 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:39:55.436+0000 7fc638ac0640 -1 osd.15 0 waiting for initial osdmap 2024-09-16T03:39:55.445 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:39:55.444+0000 7fe419cae640 -1 osd.11 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.463 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:39:55.462+0000 7fc6300bb640 -1 osd.15 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.792 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:39:55.789+0000 7fc586198640 -1 osd.13 0 waiting for initial osdmap 2024-09-16T03:39:55.794 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:39:55.791+0000 7f8f9496f640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T03:39:55.803 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:39:55.801+0000 7f8edc27b640 -1 osd.9 0 waiting for initial osdmap 2024-09-16T03:39:55.804 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:39:55.802+0000 7fc57e7ca640 -1 osd.13 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.808 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:39:55.806+0000 7f8f8cfa1640 -1 osd.5 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.811 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:39:55.809+0000 7fa9babb6640 -1 osd.12 0 waiting for initial osdmap 2024-09-16T03:39:55.813 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:39:55.811+0000 7f7e00285640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T03:39:55.817 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:39:55.814+0000 7f8ed48ad640 -1 osd.9 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.827 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:39:55.826+0000 7f7df7880640 -1 osd.0 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.827 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:39:55.826+0000 7fa9b31e8640 -1 osd.12 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:55.828 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:39:55.827+0000 7ff832e6e640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T03:39:55.846 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:39:55.845+0000 7ff82b4a0640 -1 osd.4 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:56.039 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:39:56.037+0000 7f929b303640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T03:39:56.059 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:39:56.057+0000 7f9293935640 -1 osd.3 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:39:56.629 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:39:56.901 INFO:teuthology.misc.health.smithi076.stdout: 2024-09-16T03:39:56.901 INFO:teuthology.misc.health.smithi076.stdout:{"epoch":23,"fsid":"ecf7a42a-e941-4b24-8693-0ff8b6eb41b1","created":"2024-09-16T03:39:42.686123+0000","modified":"2024-09-16T03:39:56.235015+0000","last_up_change":"2024-09-16T03:39:56.235015+0000","last_in_change":"2024-09-16T03:39:46.749797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8820e1f9-b619-44a5-abb1-7cfa97fcef00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6811","nonce":2834882614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6813","nonce":2834882614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6817","nonce":2834882614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6815","nonce":2834882614}]},"public_addr":"172.21.15.76:6811/2834882614","cluster_addr":"172.21.15.76:6813/2834882614","heartbeat_back_addr":"172.21.15.76:6817/2834882614","heartbeat_front_addr":"172.21.15.76:6815/2834882614","state":["exists","up"]},{"osd":1,"uuid":"0f58ae45-e2ff-4c4f-9875-7d34ec312cbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6809","nonce":2045143946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6811","nonce":2045143946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6815","nonce":2045143946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6813","nonce":2045143946}]},"public_addr":"172.21.15.103:6809/2045143946","cluster_addr":"172.21.15.103:6811/2045143946","heartbeat_back_addr":"172.21.15.103:6815/2045143946","heartbeat_front_addr":"172.21.15.103:6813/2045143946","state":["exists","up"]},{"osd":2,"uuid":"b369fe92-86b8-45b3-a418-024f0e4becd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6817","nonce":273257286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6819","nonce":273257286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6823","nonce":273257286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6821","nonce":273257286}]},"public_addr":"172.21.15.119:6817/273257286","cluster_addr":"172.21.15.119:6819/273257286","heartbeat_back_addr":"172.21.15.119:6823/273257286","heartbeat_front_addr":"172.21.15.119:6821/273257286","state":["exists","up"]},{"osd":3,"uuid":"b9521bf1-46ef-402c-8472-c1b476280b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6811","nonce":278480145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6815","nonce":278480145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6823","nonce":278480145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6819","nonce":278480145}]},"public_addr":"172.21.15.121:6811/278480145","cluster_addr":"172.21.15.121:6815/278480145","heartbeat_back_addr":"172.21.15.121:6823/278480145","heartbeat_front_addr":"172.21.15.121:6819/278480145","state":["exists","up"]},{"osd":4,"uuid":"15bec6a1-fc13-4822-8cfa-899b83886c10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6803","nonce":3302785479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6805","nonce":3302785479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6809","nonce":3302785479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6807","nonce":3302785479}]},"public_addr":"172.21.15.76:6803/3302785479","cluster_addr":"172.21.15.76:6805/3302785479","heartbeat_back_addr":"172.21.15.76:6809/3302785479","heartbeat_front_addr":"172.21.15.76:6807/3302785479","state":["exists","up"]},{"osd":5,"uuid":"e9b234ec-bfa4-4f72-adab-295e067b2e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6801","nonce":3765230361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6803","nonce":3765230361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6807","nonce":3765230361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6805","nonce":3765230361}]},"public_addr":"172.21.15.103:6801/3765230361","cluster_addr":"172.21.15.103:6803/3765230361","heartbeat_back_addr":"172.21.15.103:6807/3765230361","heartbeat_front_addr":"172.21.15.103:6805/3765230361","state":["exists","up"]},{"osd":6,"uuid":"6d4ae9fb-8d8f-451d-a12f-1265302b03ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6825","nonce":446165121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6827","nonce":446165121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6831","nonce":446165121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6829","nonce":446165121}]},"public_addr":"172.21.15.119:6825/446165121","cluster_addr":"172.21.15.119:6827/446165121","heartbeat_back_addr":"172.21.15.119:6831/446165121","heartbeat_front_addr":"172.21.15.119:6829/446165121","state":["exists","up"]},{"osd":7,"uuid":"2c9456ed-00ec-46aa-814e-f85eb441fc4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6810","nonce":1304791608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6814","nonce":1304791608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6822","nonce":1304791608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6818","nonce":1304791608}]},"public_addr":"172.21.15.121:6810/1304791608","cluster_addr":"172.21.15.121:6814/1304791608","heartbeat_back_addr":"172.21.15.121:6822/1304791608","heartbeat_front_addr":"172.21.15.121:6818/1304791608","state":["exists","up"]},{"osd":8,"uuid":"b618a02a-8628-48f5-bb5f-f7591aafa118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6819","nonce":2630853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6821","nonce":2630853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6825","nonce":2630853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6823","nonce":2630853906}]},"public_addr":"172.21.15.76:6819/2630853906","cluster_addr":"172.21.15.76:6821/2630853906","heartbeat_back_addr":"172.21.15.76:6825/2630853906","heartbeat_front_addr":"172.21.15.76:6823/2630853906","state":["exists","up"]},{"osd":9,"uuid":"dc3e5be7-f012-47e6-ba35-0e3354ca0c18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6817","nonce":2702548675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6819","nonce":2702548675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6823","nonce":2702548675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6821","nonce":2702548675}]},"public_addr":"172.21.15.103:6817/2702548675","cluster_addr":"172.21.15.103:6819/2702548675","heartbeat_back_addr":"172.21.15.103:6823/2702548675","heartbeat_front_addr":"172.21.15.103:6821/2702548675","state":["exists","up"]},{"osd":10,"uuid":"4e70a7a8-7ba8-42ab-a45a-e0564b561619","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6809","nonce":205142394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6811","nonce":205142394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6815","nonce":205142394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6813","nonce":205142394}]},"public_addr":"172.21.15.119:6809/205142394","cluster_addr":"172.21.15.119:6811/205142394","heartbeat_back_addr":"172.21.15.119:6815/205142394","heartbeat_front_addr":"172.21.15.119:6813/205142394","state":["exists","up"]},{"osd":11,"uuid":"d16bc493-335f-41c2-bd68-cbad93c49a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6825","nonce":1551091359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6827","nonce":1551091359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6831","nonce":1551091359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6829","nonce":1551091359}]},"public_addr":"172.21.15.121:6825/1551091359","cluster_addr":"172.21.15.121:6827/1551091359","heartbeat_back_addr":"172.21.15.121:6831/1551091359","heartbeat_front_addr":"172.21.15.121:6829/1551091359","state":["exists","up"]},{"osd":12,"uuid":"a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6827","nonce":890992293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6829","nonce":890992293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6833","nonce":890992293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6831","nonce":890992293}]},"public_addr":"172.21.15.76:6827/890992293","cluster_addr":"172.21.15.76:6829/890992293","heartbeat_back_addr":"172.21.15.76:6833/890992293","heartbeat_front_addr":"172.21.15.76:6831/890992293","state":["exists","up"]},{"osd":13,"uuid":"61e9204b-8269-4c35-b409-5cfbd5e6abbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6825","nonce":3905392668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6827","nonce":3905392668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6831","nonce":3905392668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6829","nonce":3905392668}]},"public_addr":"172.21.15.103:6825/3905392668","cluster_addr":"172.21.15.103:6827/3905392668","heartbeat_back_addr":"172.21.15.103:6831/3905392668","heartbeat_front_addr":"172.21.15.103:6829/3905392668","state":["exists","up"]},{"osd":14,"uuid":"0bd5338e-eb99-44cc-9f5f-1c8ef903d02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6801","nonce":1147733023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6803","nonce":1147733023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6807","nonce":1147733023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6805","nonce":1147733023}]},"public_addr":"172.21.15.119:6801/1147733023","cluster_addr":"172.21.15.119:6803/1147733023","heartbeat_back_addr":"172.21.15.119:6807/1147733023","heartbeat_front_addr":"172.21.15.119:6805/1147733023","state":["exists","up"]},{"osd":15,"uuid":"f240405e-d268-41cc-bd99-5a2ca71a85a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6801","nonce":3149764080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6803","nonce":3149764080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6807","nonce":3149764080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6805","nonce":3149764080}]},"public_addr":"172.21.15.121:6801/3149764080","cluster_addr":"172.21.15.121:6803/3149764080","heartbeat_back_addr":"172.21.15.121:6807/3149764080","heartbeat_front_addr":"172.21.15.121:6805/3149764080","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.541012+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.053567+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.048985+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.072636+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:39:56.912 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-16T03:39:56.912 INFO:tasks.ceph:Creating RBD pool 2024-09-16T03:39:56.912 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T03:39:57.254 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'rbd' created 2024-09-16T03:39:57.268 DEBUG:teuthology.orchestra.run.smithi076:> rbd --cluster ceph pool init rbd 2024-09-16T03:40:00.341 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T03:40:00.342 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T03:40:00.342 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T03:40:00.342 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T03:40:00.343 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T03:40:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:00.682 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":5,"flags":0,"active_gid":4106,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":296842500},{"type":"v1","addr":"172.21.15.76:6801","nonce":296842500}]},"active_addr":"172.21.15.76:6801/296842500","active_change":"2024-09-16T03:39:46.909326+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4099,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":4209041244}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":2518569325}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":3926673741}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":3869532750}]}]} 2024-09-16T03:40:00.687 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T03:40:00.687 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T03:40:00.687 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":27,"fsid":"ecf7a42a-e941-4b24-8693-0ff8b6eb41b1","created":"2024-09-16T03:39:42.686123+0000","modified":"2024-09-16T03:40:00.267105+0000","last_up_change":"2024-09-16T03:39:56.235015+0000","last_in_change":"2024-09-16T03:39:46.749797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:39:56.952065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:39:57.213209+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":27,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.0199999809265137,"score_stable":6.0199999809265137,"optimal_score":0.81000000238418579,"raw_score_acting":4.880000114440918,"raw_score_stable":4.880000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8820e1f9-b619-44a5-abb1-7cfa97fcef00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6811","nonce":2834882614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6813","nonce":2834882614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6817","nonce":2834882614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6815","nonce":2834882614}]},"public_addr":"172.21.15.76:6811/2834882614","cluster_addr":"172.21.15.76:6813/2834882614","heartbeat_back_addr":"172.21.15.76:6817/2834882614","heartbeat_front_addr":"172.21.15.76:6815/2834882614","state":["exists","up"]},{"osd":1,"uuid":"0f58ae45-e2ff-4c4f-9875-7d34ec312cbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6809","nonce":2045143946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6811","nonce":2045143946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6815","nonce":2045143946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6813","nonce":2045143946}]},"public_addr":"172.21.15.103:6809/2045143946","cluster_addr":"172.21.15.103:6811/2045143946","heartbeat_back_addr":"172.21.15.103:6815/2045143946","heartbeat_front_addr":"172.21.15.103:6813/2045143946","state":["exists","up"]},{"osd":2,"uuid":"b369fe92-86b8-45b3-a418-024f0e4becd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6817","nonce":273257286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6819","nonce":273257286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6823","nonce":273257286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6821","nonce":273257286}]},"public_addr":"172.21.15.119:6817/273257286","cluster_addr":"172.21.15.119:6819/273257286","heartbeat_back_addr":"172.21.15.119:6823/273257286","heartbeat_front_addr":"172.21.15.119:6821/273257286","state":["exists","up"]},{"osd":3,"uuid":"b9521bf1-46ef-402c-8472-c1b476280b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6811","nonce":278480145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6815","nonce":278480145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6823","nonce":278480145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6819","nonce":278480145}]},"public_addr":"172.21.15.121:6811/278480145","cluster_addr":"172.21.15.121:6815/278480145","heartbeat_back_addr":"172.21.15.121:6823/278480145","heartbeat_front_addr":"172.21.15.121:6819/278480145","state":["exists","up"]},{"osd":4,"uuid":"15bec6a1-fc13-4822-8cfa-899b83886c10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6803","nonce":3302785479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6805","nonce":3302785479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6809","nonce":3302785479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6807","nonce":3302785479}]},"public_addr":"172.21.15.76:6803/3302785479","cluster_addr":"172.21.15.76:6805/3302785479","heartbeat_back_addr":"172.21.15.76:6809/3302785479","heartbeat_front_addr":"172.21.15.76:6807/3302785479","state":["exists","up"]},{"osd":5,"uuid":"e9b234ec-bfa4-4f72-adab-295e067b2e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6801","nonce":3765230361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6803","nonce":3765230361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6807","nonce":3765230361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6805","nonce":3765230361}]},"public_addr":"172.21.15.103:6801/3765230361","cluster_addr":"172.21.15.103:6803/3765230361","heartbeat_back_addr":"172.21.15.103:6807/3765230361","heartbeat_front_addr":"172.21.15.103:6805/3765230361","state":["exists","up"]},{"osd":6,"uuid":"6d4ae9fb-8d8f-451d-a12f-1265302b03ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6825","nonce":446165121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6827","nonce":446165121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6831","nonce":446165121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6829","nonce":446165121}]},"public_addr":"172.21.15.119:6825/446165121","cluster_addr":"172.21.15.119:6827/446165121","heartbeat_back_addr":"172.21.15.119:6831/446165121","heartbeat_front_addr":"172.21.15.119:6829/446165121","state":["exists","up"]},{"osd":7,"uuid":"2c9456ed-00ec-46aa-814e-f85eb441fc4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6810","nonce":1304791608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6814","nonce":1304791608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6822","nonce":1304791608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6818","nonce":1304791608}]},"public_addr":"172.21.15.121:6810/1304791608","cluster_addr":"172.21.15.121:6814/1304791608","heartbeat_back_addr":"172.21.15.121:6822/1304791608","heartbeat_front_addr":"172.21.15.121:6818/1304791608","state":["exists","up"]},{"osd":8,"uuid":"b618a02a-8628-48f5-bb5f-f7591aafa118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6819","nonce":2630853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6821","nonce":2630853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6825","nonce":2630853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6823","nonce":2630853906}]},"public_addr":"172.21.15.76:6819/2630853906","cluster_addr":"172.21.15.76:6821/2630853906","heartbeat_back_addr":"172.21.15.76:6825/2630853906","heartbeat_front_addr":"172.21.15.76:6823/2630853906","state":["exists","up"]},{"osd":9,"uuid":"dc3e5be7-f012-47e6-ba35-0e3354ca0c18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6817","nonce":2702548675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6819","nonce":2702548675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6823","nonce":2702548675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6821","nonce":2702548675}]},"public_addr":"172.21.15.103:6817/2702548675","cluster_addr":"172.21.15.103:6819/2702548675","heartbeat_back_addr":"172.21.15.103:6823/2702548675","heartbeat_front_addr":"172.21.15.103:6821/2702548675","state":["exists","up"]},{"osd":10,"uuid":"4e70a7a8-7ba8-42ab-a45a-e0564b561619","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6809","nonce":205142394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6811","nonce":205142394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6815","nonce":205142394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6813","nonce":205142394}]},"public_addr":"172.21.15.119:6809/205142394","cluster_addr":"172.21.15.119:6811/205142394","heartbeat_back_addr":"172.21.15.119:6815/205142394","heartbeat_front_addr":"172.21.15.119:6813/205142394","state":["exists","up"]},{"osd":11,"uuid":"d16bc493-335f-41c2-bd68-cbad93c49a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6825","nonce":1551091359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6827","nonce":1551091359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6831","nonce":1551091359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6829","nonce":1551091359}]},"public_addr":"172.21.15.121:6825/1551091359","cluster_addr":"172.21.15.121:6827/1551091359","heartbeat_back_addr":"172.21.15.121:6831/1551091359","heartbeat_front_addr":"172.21.15.121:6829/1551091359","state":["exists","up"]},{"osd":12,"uuid":"a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6827","nonce":890992293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6829","nonce":890992293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6833","nonce":890992293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6831","nonce":890992293}]},"public_addr":"172.21.15.76:6827/890992293","cluster_addr":"172.21.15.76:6829/890992293","heartbeat_back_addr":"172.21.15.76:6833/890992293","heartbeat_front_addr":"172.21.15.76:6831/890992293","state":["exists","up"]},{"osd":13,"uuid":"61e9204b-8269-4c35-b409-5cfbd5e6abbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6825","nonce":3905392668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6827","nonce":3905392668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6831","nonce":3905392668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6829","nonce":3905392668}]},"public_addr":"172.21.15.103:6825/3905392668","cluster_addr":"172.21.15.103:6827/3905392668","heartbeat_back_addr":"172.21.15.103:6831/3905392668","heartbeat_front_addr":"172.21.15.103:6829/3905392668","state":["exists","up"]},{"osd":14,"uuid":"0bd5338e-eb99-44cc-9f5f-1c8ef903d02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6801","nonce":1147733023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6803","nonce":1147733023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6807","nonce":1147733023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6805","nonce":1147733023}]},"public_addr":"172.21.15.119:6801/1147733023","cluster_addr":"172.21.15.119:6803/1147733023","heartbeat_back_addr":"172.21.15.119:6807/1147733023","heartbeat_front_addr":"172.21.15.119:6805/1147733023","state":["exists","up"]},{"osd":15,"uuid":"f240405e-d268-41cc-bd99-5a2ca71a85a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6801","nonce":3149764080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6803","nonce":3149764080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6807","nonce":3149764080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6805","nonce":3149764080}]},"public_addr":"172.21.15.121:6801/3149764080","cluster_addr":"172.21.15.121:6803/3149764080","heartbeat_back_addr":"172.21.15.121:6807/3149764080","heartbeat_front_addr":"172.21.15.121:6805/3149764080","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.946705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.541012+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.279946+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.202293+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.783644+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.098081+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.053567+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.262433+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.271589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.040223+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.048985+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.049910+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.019981+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.956380+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.072636+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.047013+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:40:00.983 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T03:40:00.983 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:40:01.262 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:01.262 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":27,"fsid":"ecf7a42a-e941-4b24-8693-0ff8b6eb41b1","created":"2024-09-16T03:39:42.686123+0000","modified":"2024-09-16T03:40:00.267105+0000","last_up_change":"2024-09-16T03:39:56.235015+0000","last_in_change":"2024-09-16T03:39:46.749797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:39:56.952065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:39:57.213209+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":27,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.0199999809265137,"score_stable":6.0199999809265137,"optimal_score":0.81000000238418579,"raw_score_acting":4.880000114440918,"raw_score_stable":4.880000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8820e1f9-b619-44a5-abb1-7cfa97fcef00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6811","nonce":2834882614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6813","nonce":2834882614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6817","nonce":2834882614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6815","nonce":2834882614}]},"public_addr":"172.21.15.76:6811/2834882614","cluster_addr":"172.21.15.76:6813/2834882614","heartbeat_back_addr":"172.21.15.76:6817/2834882614","heartbeat_front_addr":"172.21.15.76:6815/2834882614","state":["exists","up"]},{"osd":1,"uuid":"0f58ae45-e2ff-4c4f-9875-7d34ec312cbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6809","nonce":2045143946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6811","nonce":2045143946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6815","nonce":2045143946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6813","nonce":2045143946}]},"public_addr":"172.21.15.103:6809/2045143946","cluster_addr":"172.21.15.103:6811/2045143946","heartbeat_back_addr":"172.21.15.103:6815/2045143946","heartbeat_front_addr":"172.21.15.103:6813/2045143946","state":["exists","up"]},{"osd":2,"uuid":"b369fe92-86b8-45b3-a418-024f0e4becd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6817","nonce":273257286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6819","nonce":273257286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6823","nonce":273257286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6821","nonce":273257286}]},"public_addr":"172.21.15.119:6817/273257286","cluster_addr":"172.21.15.119:6819/273257286","heartbeat_back_addr":"172.21.15.119:6823/273257286","heartbeat_front_addr":"172.21.15.119:6821/273257286","state":["exists","up"]},{"osd":3,"uuid":"b9521bf1-46ef-402c-8472-c1b476280b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6811","nonce":278480145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6815","nonce":278480145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6823","nonce":278480145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6819","nonce":278480145}]},"public_addr":"172.21.15.121:6811/278480145","cluster_addr":"172.21.15.121:6815/278480145","heartbeat_back_addr":"172.21.15.121:6823/278480145","heartbeat_front_addr":"172.21.15.121:6819/278480145","state":["exists","up"]},{"osd":4,"uuid":"15bec6a1-fc13-4822-8cfa-899b83886c10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6803","nonce":3302785479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6805","nonce":3302785479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6809","nonce":3302785479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6807","nonce":3302785479}]},"public_addr":"172.21.15.76:6803/3302785479","cluster_addr":"172.21.15.76:6805/3302785479","heartbeat_back_addr":"172.21.15.76:6809/3302785479","heartbeat_front_addr":"172.21.15.76:6807/3302785479","state":["exists","up"]},{"osd":5,"uuid":"e9b234ec-bfa4-4f72-adab-295e067b2e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6801","nonce":3765230361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6803","nonce":3765230361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6807","nonce":3765230361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6805","nonce":3765230361}]},"public_addr":"172.21.15.103:6801/3765230361","cluster_addr":"172.21.15.103:6803/3765230361","heartbeat_back_addr":"172.21.15.103:6807/3765230361","heartbeat_front_addr":"172.21.15.103:6805/3765230361","state":["exists","up"]},{"osd":6,"uuid":"6d4ae9fb-8d8f-451d-a12f-1265302b03ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6825","nonce":446165121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6827","nonce":446165121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6831","nonce":446165121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6829","nonce":446165121}]},"public_addr":"172.21.15.119:6825/446165121","cluster_addr":"172.21.15.119:6827/446165121","heartbeat_back_addr":"172.21.15.119:6831/446165121","heartbeat_front_addr":"172.21.15.119:6829/446165121","state":["exists","up"]},{"osd":7,"uuid":"2c9456ed-00ec-46aa-814e-f85eb441fc4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6810","nonce":1304791608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6814","nonce":1304791608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6822","nonce":1304791608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6818","nonce":1304791608}]},"public_addr":"172.21.15.121:6810/1304791608","cluster_addr":"172.21.15.121:6814/1304791608","heartbeat_back_addr":"172.21.15.121:6822/1304791608","heartbeat_front_addr":"172.21.15.121:6818/1304791608","state":["exists","up"]},{"osd":8,"uuid":"b618a02a-8628-48f5-bb5f-f7591aafa118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6819","nonce":2630853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6821","nonce":2630853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6825","nonce":2630853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6823","nonce":2630853906}]},"public_addr":"172.21.15.76:6819/2630853906","cluster_addr":"172.21.15.76:6821/2630853906","heartbeat_back_addr":"172.21.15.76:6825/2630853906","heartbeat_front_addr":"172.21.15.76:6823/2630853906","state":["exists","up"]},{"osd":9,"uuid":"dc3e5be7-f012-47e6-ba35-0e3354ca0c18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6817","nonce":2702548675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6819","nonce":2702548675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6823","nonce":2702548675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6821","nonce":2702548675}]},"public_addr":"172.21.15.103:6817/2702548675","cluster_addr":"172.21.15.103:6819/2702548675","heartbeat_back_addr":"172.21.15.103:6823/2702548675","heartbeat_front_addr":"172.21.15.103:6821/2702548675","state":["exists","up"]},{"osd":10,"uuid":"4e70a7a8-7ba8-42ab-a45a-e0564b561619","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6809","nonce":205142394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6811","nonce":205142394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6815","nonce":205142394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6813","nonce":205142394}]},"public_addr":"172.21.15.119:6809/205142394","cluster_addr":"172.21.15.119:6811/205142394","heartbeat_back_addr":"172.21.15.119:6815/205142394","heartbeat_front_addr":"172.21.15.119:6813/205142394","state":["exists","up"]},{"osd":11,"uuid":"d16bc493-335f-41c2-bd68-cbad93c49a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6825","nonce":1551091359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6827","nonce":1551091359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6831","nonce":1551091359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6829","nonce":1551091359}]},"public_addr":"172.21.15.121:6825/1551091359","cluster_addr":"172.21.15.121:6827/1551091359","heartbeat_back_addr":"172.21.15.121:6831/1551091359","heartbeat_front_addr":"172.21.15.121:6829/1551091359","state":["exists","up"]},{"osd":12,"uuid":"a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6827","nonce":890992293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6829","nonce":890992293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6833","nonce":890992293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6831","nonce":890992293}]},"public_addr":"172.21.15.76:6827/890992293","cluster_addr":"172.21.15.76:6829/890992293","heartbeat_back_addr":"172.21.15.76:6833/890992293","heartbeat_front_addr":"172.21.15.76:6831/890992293","state":["exists","up"]},{"osd":13,"uuid":"61e9204b-8269-4c35-b409-5cfbd5e6abbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6825","nonce":3905392668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6827","nonce":3905392668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6831","nonce":3905392668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3905392668},{"type":"v1","addr":"172.21.15.103:6829","nonce":3905392668}]},"public_addr":"172.21.15.103:6825/3905392668","cluster_addr":"172.21.15.103:6827/3905392668","heartbeat_back_addr":"172.21.15.103:6831/3905392668","heartbeat_front_addr":"172.21.15.103:6829/3905392668","state":["exists","up"]},{"osd":14,"uuid":"0bd5338e-eb99-44cc-9f5f-1c8ef903d02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6801","nonce":1147733023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6803","nonce":1147733023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6807","nonce":1147733023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6805","nonce":1147733023}]},"public_addr":"172.21.15.119:6801/1147733023","cluster_addr":"172.21.15.119:6803/1147733023","heartbeat_back_addr":"172.21.15.119:6807/1147733023","heartbeat_front_addr":"172.21.15.119:6805/1147733023","state":["exists","up"]},{"osd":15,"uuid":"f240405e-d268-41cc-bd99-5a2ca71a85a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6801","nonce":3149764080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6803","nonce":3149764080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6807","nonce":3149764080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6805","nonce":3149764080}]},"public_addr":"172.21.15.121:6801/3149764080","cluster_addr":"172.21.15.121:6803/3149764080","heartbeat_back_addr":"172.21.15.121:6807/3149764080","heartbeat_front_addr":"172.21.15.121:6805/3149764080","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.946705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.541012+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.279946+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.202293+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.783644+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.098081+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.053567+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.262433+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.271589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.040223+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.048985+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.049910+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.019981+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.956380+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:53.072636+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.047013+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:40:01.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T03:40:01.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T03:40:01.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T03:40:01.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T03:40:01.277 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T03:40:01.277 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T03:40:01.277 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T03:40:01.280 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T03:40:01.280 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T03:40:01.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T03:40:01.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T03:40:01.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T03:40:01.282 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-16T03:40:01.282 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-16T03:40:01.283 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-16T03:40:01.283 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-16T03:40:01.672 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:01.672 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T03:40:01.700 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:01.700 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:40:01.701 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:01.702 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:40:01.708 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:01.708 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:40:01.715 INFO:teuthology.orchestra.run.smithi076.stdout:90194313220 2024-09-16T03:40:01.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T03:40:01.736 INFO:teuthology.orchestra.run.smithi076.stdout:98784247810 2024-09-16T03:40:01.737 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:40:01.746 INFO:teuthology.orchestra.run.smithi076.stdout:94489280516 2024-09-16T03:40:01.747 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T03:40:01.759 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:01.759 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T03:40:01.764 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:01.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T03:40:01.770 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:01.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T03:40:01.800 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:01.800 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:40:01.804 INFO:teuthology.orchestra.run.smithi076.stdout:94489280516 2024-09-16T03:40:01.805 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:40:01.809 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:01.809 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T03:40:01.824 INFO:teuthology.orchestra.run.smithi076.stdout:94489280515 2024-09-16T03:40:01.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T03:40:01.933 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:01.933 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T03:40:02.012 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:02.013 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T03:40:02.555 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:02.599 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.8 2024-09-16T03:40:02.662 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:02.682 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.0 2024-09-16T03:40:02.766 INFO:teuthology.orchestra.run.smithi076.stdout:90194313219 2024-09-16T03:40:02.783 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:02.785 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313219 for osd.14 2024-09-16T03:40:02.797 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:02.812 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.7 2024-09-16T03:40:02.817 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.4 2024-09-16T03:40:02.825 INFO:teuthology.orchestra.run.smithi076.stdout:94489280514 2024-09-16T03:40:02.867 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280514 for osd.6 2024-09-16T03:40:02.918 INFO:teuthology.orchestra.run.smithi076.stdout:98784247810 2024-09-16T03:40:02.921 INFO:teuthology.orchestra.run.smithi076.stdout:98784247809 2024-09-16T03:40:02.933 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:02.941 INFO:teuthology.orchestra.run.smithi076.stdout:98784247810 2024-09-16T03:40:02.941 INFO:teuthology.orchestra.run.smithi076.stdout:98784247810 2024-09-16T03:40:02.950 INFO:teuthology.orchestra.run.smithi076.stdout:98784247810 2024-09-16T03:40:02.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.12 2024-09-16T03:40:02.964 INFO:teuthology.orchestra.run.smithi076.stdout:98784247810 2024-09-16T03:40:02.964 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247810 for osd.13 2024-09-16T03:40:02.966 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247810 got 98784247809 for osd.5 2024-09-16T03:40:02.966 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247810 for osd.3 2024-09-16T03:40:02.969 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:02.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247810 for osd.2 2024-09-16T03:40:02.975 INFO:teuthology.orchestra.run.smithi076.stdout:94489280513 2024-09-16T03:40:02.976 INFO:teuthology.orchestra.run.smithi076.stdout:94489280514 2024-09-16T03:40:02.978 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247810 for osd.9 2024-09-16T03:40:02.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247810 for osd.11 2024-09-16T03:40:02.989 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.15 2024-09-16T03:40:02.993 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280513 for osd.10 2024-09-16T03:40:02.993 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280514 for osd.1 2024-09-16T03:40:03.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T03:40:03.683 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:40:03.786 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T03:40:03.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T03:40:03.818 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:40:03.868 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T03:40:03.947 INFO:teuthology.orchestra.run.smithi076.stdout:98784247813 2024-09-16T03:40:03.959 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T03:40:03.965 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T03:40:03.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:40:03.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:40:03.969 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.8 2024-09-16T03:40:03.969 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:03.975 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:40:03.979 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T03:40:03.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T03:40:03.990 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T03:40:03.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T03:40:03.994 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:40:04.019 INFO:teuthology.orchestra.run.smithi076.stdout:98784247813 2024-09-16T03:40:04.055 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.0 2024-09-16T03:40:04.055 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.374 INFO:teuthology.orchestra.run.smithi076.stdout:90194313220 2024-09-16T03:40:04.397 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.14 2024-09-16T03:40:04.397 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.466 INFO:teuthology.orchestra.run.smithi076.stdout:98784247813 2024-09-16T03:40:04.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.7 2024-09-16T03:40:04.513 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.536 INFO:teuthology.orchestra.run.smithi076.stdout:98784247813 2024-09-16T03:40:04.577 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.4 2024-09-16T03:40:04.577 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.753 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:04.765 INFO:teuthology.orchestra.run.smithi076.stdout:94489280516 2024-09-16T03:40:04.788 INFO:teuthology.orchestra.run.smithi076.stdout:98784247813 2024-09-16T03:40:04.793 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247812 for osd.2 2024-09-16T03:40:04.793 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.795 INFO:teuthology.orchestra.run.smithi076.stdout:98784247813 2024-09-16T03:40:04.800 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.6 2024-09-16T03:40:04.800 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.15 2024-09-16T03:40:04.818 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.834 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.12 2024-09-16T03:40:04.834 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.838 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:04.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247812 for osd.11 2024-09-16T03:40:04.857 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.886 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:04.904 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247812 for osd.9 2024-09-16T03:40:04.904 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.907 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:04.919 INFO:teuthology.orchestra.run.smithi076.stdout:94489280515 2024-09-16T03:40:04.921 INFO:teuthology.orchestra.run.smithi076.stdout:98784247811 2024-09-16T03:40:04.922 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.3 2024-09-16T03:40:04.923 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.937 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.10 2024-09-16T03:40:04.937 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.938 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247810 got 98784247811 for osd.5 2024-09-16T03:40:04.939 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.943 INFO:teuthology.orchestra.run.smithi076.stdout:94489280516 2024-09-16T03:40:04.948 INFO:teuthology.orchestra.run.smithi076.stdout:98784247812 2024-09-16T03:40:04.957 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.1 2024-09-16T03:40:04.957 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.963 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247812 for osd.13 2024-09-16T03:40:04.963 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:04.963 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T03:40:04.963 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:05.277 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:05.278 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:40:05.290 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-16T03:40:04.916468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":438356,"kb_used_data":5320,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509511084,"statfs":{"total":1546188226560,"available":1545739350016,"internally_reserved":0,"allocated":5447680,"data_stored":2780204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25518,"internal_metadata":442473554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000412"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.334076+0000","last_change":"2024-09-16T03:40:00.334076+0000","last_active":"2024-09-16T03:40:00.334076+0000","last_peered":"2024-09-16T03:40:00.334076+0000","last_clean":"2024-09-16T03:40:00.334076+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:40:00.334076+0000","last_undegraded":"2024-09-16T03:40:00.334076+0000","last_fullsized":"2024-09-16T03:40:00.334076+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.276919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.336824+0000","last_change":"2024-09-16T03:40:00.337011+0000","last_active":"2024-09-16T03:40:00.336824+0000","last_peered":"2024-09-16T03:40:00.336824+0000","last_clean":"2024-09-16T03:40:00.336824+0000","last_became_active":"2024-09-16T03:39:59.086852+0000","last_became_peered":"2024-09-16T03:39:59.086852+0000","last_unstale":"2024-09-16T03:40:00.336824+0000","last_undegraded":"2024-09-16T03:40:00.336824+0000","last_fullsized":"2024-09-16T03:40:00.336824+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.961025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031859299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.333609+0000","last_change":"2024-09-16T03:40:00.333609+0000","last_active":"2024-09-16T03:40:00.333609+0000","last_peered":"2024-09-16T03:40:00.333609+0000","last_clean":"2024-09-16T03:40:00.333609+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:40:00.333609+0000","last_undegraded":"2024-09-16T03:40:00.333609+0000","last_fullsized":"2024-09-16T03:40:00.333609+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.665433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.115542+0000","last_change":"2024-09-16T03:40:01.115542+0000","last_active":"2024-09-16T03:40:01.115542+0000","last_peered":"2024-09-16T03:40:01.115542+0000","last_clean":"2024-09-16T03:40:01.115542+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:40:01.115542+0000","last_undegraded":"2024-09-16T03:40:01.115542+0000","last_fullsized":"2024-09-16T03:40:01.115542+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.045739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"27'2","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.335729+0000","last_change":"2024-09-16T03:40:00.334487+0000","last_active":"2024-09-16T03:40:00.335729+0000","last_peered":"2024-09-16T03:40:00.335729+0000","last_clean":"2024-09-16T03:40:00.335729+0000","last_became_active":"2024-09-16T03:39:58.564548+0000","last_became_peered":"2024-09-16T03:39:58.564548+0000","last_unstale":"2024-09-16T03:40:00.335729+0000","last_undegraded":"2024-09-16T03:40:00.335729+0000","last_fullsized":"2024-09-16T03:40:00.335729+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:24.220303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965529+0000","last_change":"2024-09-16T03:40:00.965529+0000","last_active":"2024-09-16T03:40:00.965529+0000","last_peered":"2024-09-16T03:40:00.965529+0000","last_clean":"2024-09-16T03:40:00.965529+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:40:00.965529+0000","last_undegraded":"2024-09-16T03:40:00.965529+0000","last_fullsized":"2024-09-16T03:40:00.965529+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.759480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.334268+0000","last_change":"2024-09-16T03:40:00.334268+0000","last_active":"2024-09-16T03:40:00.334268+0000","last_peered":"2024-09-16T03:40:00.334268+0000","last_clean":"2024-09-16T03:40:00.334268+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:40:00.334268+0000","last_undegraded":"2024-09-16T03:40:00.334268+0000","last_fullsized":"2024-09-16T03:40:00.334268+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:16.834552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"25'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965641+0000","last_change":"2024-09-16T03:40:00.965641+0000","last_active":"2024-09-16T03:40:00.965641+0000","last_peered":"2024-09-16T03:40:00.965641+0000","last_clean":"2024-09-16T03:40:00.965641+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:40:00.965641+0000","last_undegraded":"2024-09-16T03:40:00.965641+0000","last_fullsized":"2024-09-16T03:40:00.965641+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:22.133279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965103+0000","last_change":"2024-09-16T03:39:58.261761+0000","last_active":"2024-09-16T03:40:00.965103+0000","last_peered":"2024-09-16T03:40:00.965103+0000","last_clean":"2024-09-16T03:40:00.965103+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:40:00.965103+0000","last_undegraded":"2024-09-16T03:40:00.965103+0000","last_fullsized":"2024-09-16T03:40:00.965103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.270453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27308,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,7,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":258048,"data_stored":98695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":108260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[0,1,3,6,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247811,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":806912,"data_stored":637169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":108260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28204,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343636,"statfs":{"total":96636764160,"available":96607883264,"internally_reserved":0,"allocated":806912,"data_stored":637169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,5,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:05.291 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:05.567 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:40:05.579 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-16T03:40:04.916468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":438356,"kb_used_data":5320,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509511084,"statfs":{"total":1546188226560,"available":1545739350016,"internally_reserved":0,"allocated":5447680,"data_stored":2780204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25518,"internal_metadata":442473554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000412"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.334076+0000","last_change":"2024-09-16T03:40:00.334076+0000","last_active":"2024-09-16T03:40:00.334076+0000","last_peered":"2024-09-16T03:40:00.334076+0000","last_clean":"2024-09-16T03:40:00.334076+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:40:00.334076+0000","last_undegraded":"2024-09-16T03:40:00.334076+0000","last_fullsized":"2024-09-16T03:40:00.334076+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.276919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.336824+0000","last_change":"2024-09-16T03:40:00.337011+0000","last_active":"2024-09-16T03:40:00.336824+0000","last_peered":"2024-09-16T03:40:00.336824+0000","last_clean":"2024-09-16T03:40:00.336824+0000","last_became_active":"2024-09-16T03:39:59.086852+0000","last_became_peered":"2024-09-16T03:39:59.086852+0000","last_unstale":"2024-09-16T03:40:00.336824+0000","last_undegraded":"2024-09-16T03:40:00.336824+0000","last_fullsized":"2024-09-16T03:40:00.336824+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.961025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031859299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.333609+0000","last_change":"2024-09-16T03:40:00.333609+0000","last_active":"2024-09-16T03:40:00.333609+0000","last_peered":"2024-09-16T03:40:00.333609+0000","last_clean":"2024-09-16T03:40:00.333609+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:40:00.333609+0000","last_undegraded":"2024-09-16T03:40:00.333609+0000","last_fullsized":"2024-09-16T03:40:00.333609+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.665433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.115542+0000","last_change":"2024-09-16T03:40:01.115542+0000","last_active":"2024-09-16T03:40:01.115542+0000","last_peered":"2024-09-16T03:40:01.115542+0000","last_clean":"2024-09-16T03:40:01.115542+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:40:01.115542+0000","last_undegraded":"2024-09-16T03:40:01.115542+0000","last_fullsized":"2024-09-16T03:40:01.115542+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.045739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"27'2","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.335729+0000","last_change":"2024-09-16T03:40:00.334487+0000","last_active":"2024-09-16T03:40:00.335729+0000","last_peered":"2024-09-16T03:40:00.335729+0000","last_clean":"2024-09-16T03:40:00.335729+0000","last_became_active":"2024-09-16T03:39:58.564548+0000","last_became_peered":"2024-09-16T03:39:58.564548+0000","last_unstale":"2024-09-16T03:40:00.335729+0000","last_undegraded":"2024-09-16T03:40:00.335729+0000","last_fullsized":"2024-09-16T03:40:00.335729+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:24.220303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965529+0000","last_change":"2024-09-16T03:40:00.965529+0000","last_active":"2024-09-16T03:40:00.965529+0000","last_peered":"2024-09-16T03:40:00.965529+0000","last_clean":"2024-09-16T03:40:00.965529+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:40:00.965529+0000","last_undegraded":"2024-09-16T03:40:00.965529+0000","last_fullsized":"2024-09-16T03:40:00.965529+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.759480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.334268+0000","last_change":"2024-09-16T03:40:00.334268+0000","last_active":"2024-09-16T03:40:00.334268+0000","last_peered":"2024-09-16T03:40:00.334268+0000","last_clean":"2024-09-16T03:40:00.334268+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:40:00.334268+0000","last_undegraded":"2024-09-16T03:40:00.334268+0000","last_fullsized":"2024-09-16T03:40:00.334268+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:16.834552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"25'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965641+0000","last_change":"2024-09-16T03:40:00.965641+0000","last_active":"2024-09-16T03:40:00.965641+0000","last_peered":"2024-09-16T03:40:00.965641+0000","last_clean":"2024-09-16T03:40:00.965641+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:40:00.965641+0000","last_undegraded":"2024-09-16T03:40:00.965641+0000","last_fullsized":"2024-09-16T03:40:00.965641+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:22.133279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965103+0000","last_change":"2024-09-16T03:39:58.261761+0000","last_active":"2024-09-16T03:40:00.965103+0000","last_peered":"2024-09-16T03:40:00.965103+0000","last_clean":"2024-09-16T03:40:00.965103+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:40:00.965103+0000","last_undegraded":"2024-09-16T03:40:00.965103+0000","last_fullsized":"2024-09-16T03:40:00.965103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.270453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27308,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,7,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":258048,"data_stored":98695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":108260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[0,1,3,6,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247811,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":806912,"data_stored":637169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":108260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28204,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343636,"statfs":{"total":96636764160,"available":96607883264,"internally_reserved":0,"allocated":806912,"data_stored":637169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,5,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:05.580 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T03:40:05.580 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T03:40:05.580 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T03:40:05.580 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T03:40:05.891 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:05.892 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T03:40:05.904 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T03:40:05.905 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T03:40:05.915 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T03:40:05.916 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}} 2024-09-16T03:40:05.916 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-16T03:40:05.916 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-16T03:40:06.212 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T03:40:06.212 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T03:40:06.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:06.494 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:40:06.506 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T03:40:06.464691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":438368,"kb_used_data":5336,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509511072,"statfs":{"total":1546188226560,"available":1545739337728,"internally_reserved":0,"allocated":5464064,"data_stored":2789750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25518,"internal_metadata":442473554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.548634"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.334076+0000","last_change":"2024-09-16T03:40:00.334076+0000","last_active":"2024-09-16T03:40:00.334076+0000","last_peered":"2024-09-16T03:40:00.334076+0000","last_clean":"2024-09-16T03:40:00.334076+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:40:00.334076+0000","last_undegraded":"2024-09-16T03:40:00.334076+0000","last_fullsized":"2024-09-16T03:40:00.334076+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.276919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.336824+0000","last_change":"2024-09-16T03:40:00.337011+0000","last_active":"2024-09-16T03:40:00.336824+0000","last_peered":"2024-09-16T03:40:00.336824+0000","last_clean":"2024-09-16T03:40:00.336824+0000","last_became_active":"2024-09-16T03:39:59.086852+0000","last_became_peered":"2024-09-16T03:39:59.086852+0000","last_unstale":"2024-09-16T03:40:00.336824+0000","last_undegraded":"2024-09-16T03:40:00.336824+0000","last_fullsized":"2024-09-16T03:40:00.336824+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.961025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031859299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.333609+0000","last_change":"2024-09-16T03:40:00.333609+0000","last_active":"2024-09-16T03:40:00.333609+0000","last_peered":"2024-09-16T03:40:00.333609+0000","last_clean":"2024-09-16T03:40:00.333609+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:40:00.333609+0000","last_undegraded":"2024-09-16T03:40:00.333609+0000","last_fullsized":"2024-09-16T03:40:00.333609+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.665433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.115542+0000","last_change":"2024-09-16T03:40:01.115542+0000","last_active":"2024-09-16T03:40:01.115542+0000","last_peered":"2024-09-16T03:40:01.115542+0000","last_clean":"2024-09-16T03:40:01.115542+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:40:01.115542+0000","last_undegraded":"2024-09-16T03:40:01.115542+0000","last_fullsized":"2024-09-16T03:40:01.115542+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.045739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"27'2","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.335729+0000","last_change":"2024-09-16T03:40:00.334487+0000","last_active":"2024-09-16T03:40:00.335729+0000","last_peered":"2024-09-16T03:40:00.335729+0000","last_clean":"2024-09-16T03:40:00.335729+0000","last_became_active":"2024-09-16T03:39:58.564548+0000","last_became_peered":"2024-09-16T03:39:58.564548+0000","last_unstale":"2024-09-16T03:40:00.335729+0000","last_undegraded":"2024-09-16T03:40:00.335729+0000","last_fullsized":"2024-09-16T03:40:00.335729+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:24.220303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965529+0000","last_change":"2024-09-16T03:40:00.965529+0000","last_active":"2024-09-16T03:40:00.965529+0000","last_peered":"2024-09-16T03:40:00.965529+0000","last_clean":"2024-09-16T03:40:00.965529+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:40:00.965529+0000","last_undegraded":"2024-09-16T03:40:00.965529+0000","last_fullsized":"2024-09-16T03:40:00.965529+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.759480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.334268+0000","last_change":"2024-09-16T03:40:00.334268+0000","last_active":"2024-09-16T03:40:00.334268+0000","last_peered":"2024-09-16T03:40:00.334268+0000","last_clean":"2024-09-16T03:40:00.334268+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:40:00.334268+0000","last_undegraded":"2024-09-16T03:40:00.334268+0000","last_fullsized":"2024-09-16T03:40:00.334268+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:16.834552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"25'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965641+0000","last_change":"2024-09-16T03:40:00.965641+0000","last_active":"2024-09-16T03:40:00.965641+0000","last_peered":"2024-09-16T03:40:00.965641+0000","last_clean":"2024-09-16T03:40:00.965641+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:40:00.965641+0000","last_undegraded":"2024-09-16T03:40:00.965641+0000","last_fullsized":"2024-09-16T03:40:00.965641+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:22.133279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965103+0000","last_change":"2024-09-16T03:39:58.261761+0000","last_active":"2024-09-16T03:40:00.965103+0000","last_peered":"2024-09-16T03:40:00.965103+0000","last_clean":"2024-09-16T03:40:00.965103+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:40:00.965103+0000","last_undegraded":"2024-09-16T03:40:00.965103+0000","last_fullsized":"2024-09-16T03:40:00.965103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.270453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27308,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,7,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":108260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[0,1,3,6,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247811,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":806912,"data_stored":637169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":108260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28204,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343636,"statfs":{"total":96636764160,"available":96607883264,"internally_reserved":0,"allocated":806912,"data_stored":637169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,5,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:06.507 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:06.786 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:06.787 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:40:06.798 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T03:40:06.464691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":438368,"kb_used_data":5336,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509511072,"statfs":{"total":1546188226560,"available":1545739337728,"internally_reserved":0,"allocated":5464064,"data_stored":2789750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25518,"internal_metadata":442473554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.548634"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.334076+0000","last_change":"2024-09-16T03:40:00.334076+0000","last_active":"2024-09-16T03:40:00.334076+0000","last_peered":"2024-09-16T03:40:00.334076+0000","last_clean":"2024-09-16T03:40:00.334076+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:40:00.334076+0000","last_undegraded":"2024-09-16T03:40:00.334076+0000","last_fullsized":"2024-09-16T03:40:00.334076+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.276919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.336824+0000","last_change":"2024-09-16T03:40:00.337011+0000","last_active":"2024-09-16T03:40:00.336824+0000","last_peered":"2024-09-16T03:40:00.336824+0000","last_clean":"2024-09-16T03:40:00.336824+0000","last_became_active":"2024-09-16T03:39:59.086852+0000","last_became_peered":"2024-09-16T03:39:59.086852+0000","last_unstale":"2024-09-16T03:40:00.336824+0000","last_undegraded":"2024-09-16T03:40:00.336824+0000","last_fullsized":"2024-09-16T03:40:00.336824+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.961025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031859299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.333609+0000","last_change":"2024-09-16T03:40:00.333609+0000","last_active":"2024-09-16T03:40:00.333609+0000","last_peered":"2024-09-16T03:40:00.333609+0000","last_clean":"2024-09-16T03:40:00.333609+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:40:00.333609+0000","last_undegraded":"2024-09-16T03:40:00.333609+0000","last_fullsized":"2024-09-16T03:40:00.333609+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.665433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.115542+0000","last_change":"2024-09-16T03:40:01.115542+0000","last_active":"2024-09-16T03:40:01.115542+0000","last_peered":"2024-09-16T03:40:01.115542+0000","last_clean":"2024-09-16T03:40:01.115542+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:40:01.115542+0000","last_undegraded":"2024-09-16T03:40:01.115542+0000","last_fullsized":"2024-09-16T03:40:01.115542+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.045739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"27'2","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.335729+0000","last_change":"2024-09-16T03:40:00.334487+0000","last_active":"2024-09-16T03:40:00.335729+0000","last_peered":"2024-09-16T03:40:00.335729+0000","last_clean":"2024-09-16T03:40:00.335729+0000","last_became_active":"2024-09-16T03:39:58.564548+0000","last_became_peered":"2024-09-16T03:39:58.564548+0000","last_unstale":"2024-09-16T03:40:00.335729+0000","last_undegraded":"2024-09-16T03:40:00.335729+0000","last_fullsized":"2024-09-16T03:40:00.335729+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:24.220303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965529+0000","last_change":"2024-09-16T03:40:00.965529+0000","last_active":"2024-09-16T03:40:00.965529+0000","last_peered":"2024-09-16T03:40:00.965529+0000","last_clean":"2024-09-16T03:40:00.965529+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:40:00.965529+0000","last_undegraded":"2024-09-16T03:40:00.965529+0000","last_fullsized":"2024-09-16T03:40:00.965529+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.759480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.334268+0000","last_change":"2024-09-16T03:40:00.334268+0000","last_active":"2024-09-16T03:40:00.334268+0000","last_peered":"2024-09-16T03:40:00.334268+0000","last_clean":"2024-09-16T03:40:00.334268+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:40:00.334268+0000","last_undegraded":"2024-09-16T03:40:00.334268+0000","last_fullsized":"2024-09-16T03:40:00.334268+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:16.834552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"25'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965641+0000","last_change":"2024-09-16T03:40:00.965641+0000","last_active":"2024-09-16T03:40:00.965641+0000","last_peered":"2024-09-16T03:40:00.965641+0000","last_clean":"2024-09-16T03:40:00.965641+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:40:00.965641+0000","last_undegraded":"2024-09-16T03:40:00.965641+0000","last_fullsized":"2024-09-16T03:40:00.965641+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:22.133279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-16T03:40:00.965103+0000","last_change":"2024-09-16T03:39:58.261761+0000","last_active":"2024-09-16T03:40:00.965103+0000","last_peered":"2024-09-16T03:40:00.965103+0000","last_clean":"2024-09-16T03:40:00.965103+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:40:00.965103+0000","last_undegraded":"2024-09-16T03:40:00.965103+0000","last_fullsized":"2024-09-16T03:40:00.965103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.270453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27308,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,7,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":108260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[0,1,3,6,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247811,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":806912,"data_stored":637169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":108260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":274432,"data_stored":108241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28204,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343636,"statfs":{"total":96636764160,"available":96607883264,"internally_reserved":0,"allocated":806912,"data_stored":637169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,5,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:06.799 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T03:40:06.800 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T03:40:07.080 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 28 2024-09-16T03:40:07.080 INFO:teuthology.orchestra.run.smithi076.stdout:fsid ecf7a42a-e941-4b24-8693-0ff8b6eb41b1 2024-09-16T03:40:07.080 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-09-16T03:39:42.686123+0000 2024-09-16T03:40:07.080 INFO:teuthology.orchestra.run.smithi076.stdout:modified 2024-09-16T03:40:06.460322+0000 2024-09-16T03:40:07.080 INFO:teuthology.orchestra.run.smithi076.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T03:40:07.080 INFO:teuthology.orchestra.run.smithi076.stdout:crush_version 6 2024-09-16T03:40:07.080 INFO:teuthology.orchestra.run.smithi076.stdout:full_ratio 0.95 2024-09-16T03:40:07.080 INFO:teuthology.orchestra.run.smithi076.stdout:backfillfull_ratio 0.9 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:nearfull_ratio 0.85 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:require_min_compat_client luminous 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:min_compat_client jewel 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:require_osd_release squid 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:stretch_mode_enabled false 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 26 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 27 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 6.02 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:max_osd 16 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6810/2834882614,v1:172.21.15.76:6811/2834882614] [v2:172.21.15.76:6812/2834882614,v1:172.21.15.76:6813/2834882614] exists,up 8820e1f9-b619-44a5-abb1-7cfa97fcef00 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6808/2045143946,v1:172.21.15.103:6809/2045143946] [v2:172.21.15.103:6810/2045143946,v1:172.21.15.103:6811/2045143946] exists,up 0f58ae45-e2ff-4c4f-9875-7d34ec312cbe 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6816/273257286,v1:172.21.15.119:6817/273257286] [v2:172.21.15.119:6818/273257286,v1:172.21.15.119:6819/273257286] exists,up b369fe92-86b8-45b3-a418-024f0e4becd7 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6809/278480145,v1:172.21.15.121:6811/278480145] [v2:172.21.15.121:6813/278480145,v1:172.21.15.121:6815/278480145] exists,up b9521bf1-46ef-402c-8472-c1b476280b23 2024-09-16T03:40:07.081 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6802/3302785479,v1:172.21.15.76:6803/3302785479] [v2:172.21.15.76:6804/3302785479,v1:172.21.15.76:6805/3302785479] exists,up 15bec6a1-fc13-4822-8cfa-899b83886c10 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6800/3765230361,v1:172.21.15.103:6801/3765230361] [v2:172.21.15.103:6802/3765230361,v1:172.21.15.103:6803/3765230361] exists,up e9b234ec-bfa4-4f72-adab-295e067b2e61 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6 up in weight 1 up_from 22 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6824/446165121,v1:172.21.15.119:6825/446165121] [v2:172.21.15.119:6826/446165121,v1:172.21.15.119:6827/446165121] exists,up 6d4ae9fb-8d8f-451d-a12f-1265302b03ab 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6808/1304791608,v1:172.21.15.121:6810/1304791608] [v2:172.21.15.121:6812/1304791608,v1:172.21.15.121:6814/1304791608] exists,up 2c9456ed-00ec-46aa-814e-f85eb441fc4f 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8 up in weight 1 up_from 23 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6818/2630853906,v1:172.21.15.76:6819/2630853906] [v2:172.21.15.76:6820/2630853906,v1:172.21.15.76:6821/2630853906] exists,up b618a02a-8628-48f5-bb5f-f7591aafa118 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9 up in weight 1 up_from 23 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6816/2702548675,v1:172.21.15.103:6817/2702548675] [v2:172.21.15.103:6818/2702548675,v1:172.21.15.103:6819/2702548675] exists,up dc3e5be7-f012-47e6-ba35-0e3354ca0c18 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6808/205142394,v1:172.21.15.119:6809/205142394] [v2:172.21.15.119:6810/205142394,v1:172.21.15.119:6811/205142394] exists,up 4e70a7a8-7ba8-42ab-a45a-e0564b561619 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11 up in weight 1 up_from 23 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6824/1551091359,v1:172.21.15.121:6825/1551091359] [v2:172.21.15.121:6826/1551091359,v1:172.21.15.121:6827/1551091359] exists,up d16bc493-335f-41c2-bd68-cbad93c49a5d 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6826/890992293,v1:172.21.15.76:6827/890992293] [v2:172.21.15.76:6828/890992293,v1:172.21.15.76:6829/890992293] exists,up a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6824/3905392668,v1:172.21.15.103:6825/3905392668] [v2:172.21.15.103:6826/3905392668,v1:172.21.15.103:6827/3905392668] exists,up 61e9204b-8269-4c35-b409-5cfbd5e6abbb 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14 up in weight 1 up_from 21 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6800/1147733023,v1:172.21.15.119:6801/1147733023] [v2:172.21.15.119:6802/1147733023,v1:172.21.15.119:6803/1147733023] exists,up 0bd5338e-eb99-44cc-9f5f-1c8ef903d02c 2024-09-16T03:40:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15 up in weight 1 up_from 23 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] [v2:172.21.15.121:6802/3149764080,v1:172.21.15.121:6803/3149764080] exists,up f240405e-d268-41cc-bd99-5a2ca71a85a9 2024-09-16T03:40:07.093 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6810/2834882614,v1:172.21.15.76:6811/2834882614] [v2:172.21.15.76:6812/2834882614,v1:172.21.15.76:6813/2834882614] exists,up 8820e1f9-b619-44a5-abb1-7cfa97fcef00', 'osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6808/2045143946,v1:172.21.15.103:6809/2045143946] [v2:172.21.15.103:6810/2045143946,v1:172.21.15.103:6811/2045143946] exists,up 0f58ae45-e2ff-4c4f-9875-7d34ec312cbe', 'osd.2 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6816/273257286,v1:172.21.15.119:6817/273257286] [v2:172.21.15.119:6818/273257286,v1:172.21.15.119:6819/273257286] exists,up b369fe92-86b8-45b3-a418-024f0e4becd7', 'osd.3 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6809/278480145,v1:172.21.15.121:6811/278480145] [v2:172.21.15.121:6813/278480145,v1:172.21.15.121:6815/278480145] exists,up b9521bf1-46ef-402c-8472-c1b476280b23', 'osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6802/3302785479,v1:172.21.15.76:6803/3302785479] [v2:172.21.15.76:6804/3302785479,v1:172.21.15.76:6805/3302785479] exists,up 15bec6a1-fc13-4822-8cfa-899b83886c10', 'osd.5 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6800/3765230361,v1:172.21.15.103:6801/3765230361] [v2:172.21.15.103:6802/3765230361,v1:172.21.15.103:6803/3765230361] exists,up e9b234ec-bfa4-4f72-adab-295e067b2e61', 'osd.6 up in weight 1 up_from 22 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6824/446165121,v1:172.21.15.119:6825/446165121] [v2:172.21.15.119:6826/446165121,v1:172.21.15.119:6827/446165121] exists,up 6d4ae9fb-8d8f-451d-a12f-1265302b03ab', 'osd.7 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6808/1304791608,v1:172.21.15.121:6810/1304791608] [v2:172.21.15.121:6812/1304791608,v1:172.21.15.121:6814/1304791608] exists,up 2c9456ed-00ec-46aa-814e-f85eb441fc4f', 'osd.8 up in weight 1 up_from 23 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6818/2630853906,v1:172.21.15.76:6819/2630853906] [v2:172.21.15.76:6820/2630853906,v1:172.21.15.76:6821/2630853906] exists,up b618a02a-8628-48f5-bb5f-f7591aafa118', 'osd.9 up in weight 1 up_from 23 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6816/2702548675,v1:172.21.15.103:6817/2702548675] [v2:172.21.15.103:6818/2702548675,v1:172.21.15.103:6819/2702548675] exists,up dc3e5be7-f012-47e6-ba35-0e3354ca0c18', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6808/205142394,v1:172.21.15.119:6809/205142394] [v2:172.21.15.119:6810/205142394,v1:172.21.15.119:6811/205142394] exists,up 4e70a7a8-7ba8-42ab-a45a-e0564b561619', 'osd.11 up in weight 1 up_from 23 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6824/1551091359,v1:172.21.15.121:6825/1551091359] [v2:172.21.15.121:6826/1551091359,v1:172.21.15.121:6827/1551091359] exists,up d16bc493-335f-41c2-bd68-cbad93c49a5d', 'osd.12 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6826/890992293,v1:172.21.15.76:6827/890992293] [v2:172.21.15.76:6828/890992293,v1:172.21.15.76:6829/890992293] exists,up a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1', 'osd.13 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6824/3905392668,v1:172.21.15.103:6825/3905392668] [v2:172.21.15.103:6826/3905392668,v1:172.21.15.103:6827/3905392668] exists,up 61e9204b-8269-4c35-b409-5cfbd5e6abbb', 'osd.14 up in weight 1 up_from 21 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6800/1147733023,v1:172.21.15.119:6801/1147733023] [v2:172.21.15.119:6802/1147733023,v1:172.21.15.119:6803/1147733023] exists,up 0bd5338e-eb99-44cc-9f5f-1c8ef903d02c', 'osd.15 up in weight 1 up_from 23 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] [v2:172.21.15.121:6802/3149764080,v1:172.21.15.121:6803/3149764080] exists,up f240405e-d268-41cc-bd99-5a2ca71a85a9'] 2024-09-16T03:40:07.093 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-16T03:40:07.192 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:07.192 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mon.a", 2024-09-16T03:40:07.192 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster": "ceph", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "admin_socket_mode": "", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_client_required": "cephx, none", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_cluster_required": "cephx", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_debug": "true", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_service_required": "cephx", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_supported": "cephx", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio": "true", 2024-09-16T03:40:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio_reap_max": "16", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_async_discard": "false", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_block_size": "4096", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_aio": "false", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_enable_discard": "false", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_flock_retry": "3", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_inject_crash": "0", 2024-09-16T03:40:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_ioring": "false", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_ioring_hipri": "false", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_type": "", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_allocator": "hybrid", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_buffered_io": "true", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T03:40:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_replay_recovery": "false", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T03:40:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_sync_write": "false", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_allocator": "hybrid", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bdev_label_multi": "true", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-16T03:40:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_create": "true", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_db_create": "false", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_db_path": "", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_db_size": "0", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_path": "", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_size": "107374182400", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_wal_create": "false", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_wal_path": "", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs": "true", 2024-09-16T03:40:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_autotune": "true", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T03:40:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_size": "0", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_type": "2q", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_clone_cow": "true", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T03:40:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_mode": "none", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_freelist": "false", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T03:40:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T03:40:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fail_eio": "false", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T03:40:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T03:40:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_blob_size": "0", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T03:40:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_mem": "512", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T03:40:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_tracing": "false", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T03:40:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T03:40:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_cluster_require_version": "2", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_require_signatures": "false", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_require_version": "2", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_service_require_signatures": "false", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_service_require_version": "2", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_sign_messages": "true", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "chdir": "", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "client_acl_type": "", 2024-09-16T03:40:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "client_asio_thread_count": "2", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_cache_mid": "0.750000", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_cache_size": "16384", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_caps_release_delay": "5", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_check_pool_perm": "true", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_force_sync_read": "false", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_getattr_caps": "false", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_inject_features": "", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_die_on_failed_remount": "false", 2024-09-16T03:40:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_dirsize_rbytes": "true", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_force_lazyio": "false", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_fs": "", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_inject_release_failure": "false", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_max_inline_size": "4096", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mds_namespace": "", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_metadata": "", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mount_gid": "-1", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mount_timeout": "300", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mount_uid": "-1", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mountpoint": "/", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_notify_timeout": "10", 2024-09-16T03:40:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc": "true", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_max_objects": "1000", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_size": "209715200", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_permissions": "true", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_quota": "true", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_quota_df": "true", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_readahead_max_bytes": "0", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_readahead_max_periods": "4", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_readahead_min": "131072", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_reconnect_stale": "false", 2024-09-16T03:40:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_shutdown_timeout": "30", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_snapdir": ".snap", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_tick_interval": "1", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_trace": "", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_use_faked_inos": "false", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_use_random_mds": "false", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_graylog": "false", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_graylog_port": "12201", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_monitors": "default=true", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_syslog": "false", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T03:40:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_syslog_level": "info", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "-", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_network": "", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_network_interface": "", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zlib_isal": "false", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zlib_level": "5", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zstd_level": "1", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T03:40:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_osd_stat_interval": "0", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_seastar_cpu_cores": "", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_seastar_num_threads": "0", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location_hook": "", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location_hook_timeout": "10", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "daemonize": "false", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "daos_pool": "tank", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_alienstore": "0/5", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_asok": "1/5", 2024-09-16T03:40:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_asok_assert_abort": "false", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_auth": "20/20", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_bdev": "1/3", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_bluefs": "1/5", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_bluestore": "1/5", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_buffer": "0/1", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_cephsqlite": "0/5", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_client": "0/5", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_compressor": "1/5", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_context": "0/1", 2024-09-16T03:40:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_crimson_interrupt": "0/5", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_crush": "1/1", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_crypto": "1/5", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_cyanstore": "0/5", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_dpdk": "1/5", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_eventtrace": "1/5", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_filer": "0/1", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_filestore": "1/3", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_finisher": "1/1", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_fuse": "1/5", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T03:40:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_javaclient": "1/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_journal": "1/3", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_journaler": "0/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_kstore": "1/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_lockdep": "0/1", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mclock": "1/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds": "1/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_balancer": "1/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_locker": "1/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_log": "1/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_migrator": "1/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_quiesce": "3/5", 2024-09-16T03:40:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_memstore": "1/5", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mgr": "2/5", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mgrc": "1/5", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mon": "20/20", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_monc": "0/10", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_ms": "1/1", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_none": "0/5", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_objclass": "0/5", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_objectcacher": "0/5", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_objecter": "0/1", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_optracker": "0/5", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_osd": "1/5", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_paxos": "20/20", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_perfcounter": "1/5", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_prioritycache": "1/5", 2024-09-16T03:40:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rados": "0/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd": "0/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd_replay": "0/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_refs": "0/0", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_reserver": "1/1", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw": "1/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw_access": "1/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw_dbstore": "1/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw_flight": "1/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-16T03:40:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw_sync": "1/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rocksdb": "4/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_backref": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_cache": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_device": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_epm": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_journal": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_lba": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_odata": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_omap": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_onode": "0/5", 2024-09-16T03:40:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_t": "0/5", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_tm": "0/5", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_striper": "0/1", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_test": "0/5", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_throttle": "1/1", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_timer": "0/1", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_tp": "0/5", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_trace": "1/5", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "device_failure_prediction_mode": "none", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_graylog": "false", 2024-09-16T03:40:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_journald": "false", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_stderr": "true", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_syslog": "false", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event_tracing": "false", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_http_port": "9926", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_prio_limit": "5", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_sort_metrics": "true", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_stats_period": "5", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "fake_statfs_for_testing": "0", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "fatal_signal_handlers": "true", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "filer_max_purge_ops": "10", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "filer_max_truncate_ops": "128", 2024-09-16T03:40:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_blackhole": "false", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_btrfs_snap": "true", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_caller_concurrency": "10", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_omap_check": "false", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_verify_split": "false", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_dump_file": "", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T03:40:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fadvise": "true", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fail_eio": "true", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fd_cache_size": "128", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fiemap": "false", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_inject_stall": "0", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_journal_parallel": "false", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_journal_trailing": "false", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_journal_writeahead": "false", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_kill_at": "0", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T03:40:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_merge_threshold": "-10", 2024-09-16T03:40:07.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_odsync_write": "false", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_omap_backend_path": "", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_op_threads": "2", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_punch_hole": "false", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T03:40:07.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_ops": "50", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_seek_data_hole": "false", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_sloppy_crc": "false", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_splice": "false", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_split_multiple": "2", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_split_rand_factor": "20", 2024-09-16T03:40:07.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_update_to": "1000", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T03:40:07.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_xfs_extsize": "false", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_zfs_snap": "false", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fio_dir": "/tmp/fio", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "ecf7a42a-e941-4b24-8693-0ff8b6eb41b1", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_allow_other": "true", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_big_writes": "true", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_debug": "false", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_default_permissions": "false", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_disable_pagecache": "false", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_max_write": "0", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_multithreaded": "true", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_require_active_mds": "true", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_set_user_groups": "true", 2024-09-16T03:40:07.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_splice_move": "true", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_splice_read": "true", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_splice_write": "true", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "gss_target_name": "ceph", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_file": "", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_inject_failure": "0", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_interval": "5", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "host": "smithi076", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T03:40:07.231 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "inject_early_sigterm": "false", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "jaeger_agent_port": "6799", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "jaeger_tracing_enable": "false", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_aio": "true", 2024-09-16T03:40:07.232 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_align_min_size": "65536", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_block_align": "true", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_block_size": "4096", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_dio": "true", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_discard": "false", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_force_aio": "false", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_ignore_corruption": "false", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_max_write_entries": "100", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_replay_from": "0", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T03:40:07.233 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_write_header_frequency": "0", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_zero_on_create": "false", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "journaler_prefetch_periods": "10", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "journaler_prezero_periods": "5", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "journaler_write_head_interval": "15", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "key": "", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "keyfile": "", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_backend": "rocksdb", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_max_bytes": "67108864", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_max_ops": "512", 2024-09-16T03:40:07.234 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_onode_map_size": "1024", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_sync_transaction": "false", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "librados_thread_count": "2", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "lockdep": "false", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "lockdep_force_backtrace": "false", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "log_coarse_timestamps": "true", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "log_flush_on_exit": "false", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "log_graylog_port": "12201", 2024-09-16T03:40:07.235 INFO:teuthology.orchestra.run.smithi076.stdout: "log_max_new": "1000", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "log_max_recent": "10000", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "log_stderr_prefix": "", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": "true", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_graylog": "false", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_journald": "false", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_stderr": "false", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_syslog": "false", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_action_on_write_error": "1", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_alternate_name_max": "8192", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_asio_thread_count": "2", 2024-09-16T03:40:07.236 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_export_pin": "true", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_interval": "10", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_max": "-1", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_max_until": "-1", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_merge_size": "50", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T03:40:07.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_mode": "0", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_overload_epochs": "2", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_bits": "3", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_size": "10000", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T03:40:07.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_mid": "0.700000", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_quiesce_delay": "0", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_trim_interval": "1", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T03:40:07.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_auth_pins": "false", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_frag": "false", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_scatterstat": "false", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_subtrees": "false", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_default_dir_hash": "2", 2024-09-16T03:40:07.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_defer_session_stale": "true", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dir_max_entries": "0", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dir_prefetch": "true", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_early_reply": "true", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_enable_op_tracker": "true", 2024-09-16T03:40:07.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_enforce_unique_name": "true", 2024-09-16T03:40:07.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T03:40:07.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T03:40:07.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_health_dummy": "false", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_join_fs": "", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_journal_format": "1", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_create_at": "0", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_export_at": "0", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_import_at": "0", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_journal_at": "0", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T03:40:07.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_link_at": "0", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_openc_at": "0", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_rename_at": "0", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_shutdown_at": "0", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_event_large_threshold": "524288", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_max_events": "-1", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_max_segments": "128", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_pause": "false", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_segment_size": "0", 2024-09-16T03:40:07.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_trim_threshold": "128", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_completed_requests": "100000", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_export_size": "20971520", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_file_recover": "32", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_purge_files": "64", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_purge_ops": "8192", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T03:40:07.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_metrics_update_interval": "2", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_min_caps_per_client": "100", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_numa_node": "-1", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_history_duration": "600", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_history_size": "20", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_log_threshold": "5", 2024-09-16T03:40:07.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_ping_grace": "15", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_ping_interval": "5", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_max_caps": "30000", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_replay_interval": "1.000000", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T03:40:07.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_root_ino_gid": "0", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_root_ino_uid": "0", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_shutdown_check": "0", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_skip_ino": "0", 2024-09-16T03:40:07.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_snap_min_uid": "0", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_snap_rstat": "false", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_symlink_recovery": "true", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_thrash_exports": "0", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_thrash_fragments": "0", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_tick_interval": "5.000000", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_valgrind_exit": "false", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_verify_backtrace": "1", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_verify_scatter": "false", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_wipe_sessions": "false", 2024-09-16T03:40:07.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool_debug": "false", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_page_set": "false", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_page_size": "65536", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_client_bytes": "134217728", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_client_messages": "512", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_disabled_modules": "", 2024-09-16T03:40:07.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_enable_op_tracker": "true", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mds_messages": "128", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mon_messages": "128", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_op_history_duration": "600", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_op_history_size": "20", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-16T03:40:07.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_op_log_threshold": "5", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_osd_messages": "8192", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_pool": "true", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_standby_modules": "true", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_stats_period": "5", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_stats_threshold": "5", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_tick_period": "2", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_allow_pool_delete": "true", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_allow_pool_size_one": "true", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T03:40:07.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_bytes": "104857600", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_directed_command_retry": "5", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_target_rank": "-1", 2024-09-16T03:40:07.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_file_level": "debug", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_journald": "false", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-16T03:40:07.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_compact_on_start": "false", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_compact_on_trim": "true", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cpu_threads": "4", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data_avail_crit": "5", 2024-09-16T03:40:07.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data_avail_warn": "5", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_dump_json": "false", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_extra_checks": "false", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_require_reef": "false", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_require_squid": "false", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T03:40:07.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_down_mkfs_grace": "120", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_down_uptime_grace": "60", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_election_default_strategy": "3", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_election_timeout": "5.000000", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_enable_op_tracker": "true", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_fake_pool_delete": "false", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_force_quorum_join": "false", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T03:40:07.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_detail_to_clog": "false", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_log_update_period": "5", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_max_detail": "50", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_to_clog": "true", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_host": "172.21.15.76,172.21.15.103,172.21.15.119", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_host_override": "", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_initial_members": "", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T03:40:07.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_lease": "5.000000", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_log_full_interval": "50", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_log_max": "10000", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_log_max_summary": "50", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_log_epochs": "500", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_osd": "10000", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T03:40:07.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_memory_autotune": "true", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_memory_target": "2147483648", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_digest_period": "5", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_min_osdmap_epochs": "500", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_complaint_time": "30", 2024-09-16T03:40:07.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_duration": "600", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_size": "20", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_log_threshold": "5", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_cache_size": "500", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T03:40:07.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-16T03:40:07.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_report_timeout": "900", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T03:40:07.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T03:40:07.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_max_osds": "4", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_interval": "300", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_max_keys": "100", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_timeout": "300", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_session_timeout": "300", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_smart_report_timeout": "5", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T03:40:07.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_pool_size": "4", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_debug": "false", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_tick_interval": "5", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_use_min_delay_socket": "false", 2024-09-16T03:40:07.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_older_version": "true", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": "", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi076.stdout: "motr_tracing_enabled": "false", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_op_threads": "3", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_cm": "false", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_device_name": "", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T03:40:07.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_sl": "3", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_type": "ib", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_reap_threshold": "5", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_before_connect": "false", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_ipv4": "true", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_ipv6": "false", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_msgr1": "true", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_msgr2": "true", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_port_max": "7568", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_port_min": "6800", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T03:40:07.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_retry_count": "3", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_retry_delay": "5", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_client": "false", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_mds": "false", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_mgr": "false", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_mon": "false", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_osd": "false", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_client_mode": "crc secure", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_cluster_type": "", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_compress_secure": "false", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_crc_data": "true", 2024-09-16T03:40:07.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_crc_header": "true", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_bug": "true", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_old_message": "true", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T03:40:07.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_hugepages": "", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_lro": "true", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_pmd": "", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_port_id": "0", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dump_on_send": "false", 2024-09-16T03:40:07.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_delay_type": "", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_network_congestion": "0", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_socket_failures": "2500", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_max_accept_failures": "4", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_max_backoff": "15.000000", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T03:40:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_osd_compress_mode": "none", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_pq_min_cost": "65536", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_public_type": "", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_service_mode": "crc secure", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_nodelay": "true", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_type": "async+posix", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "no_config_file": "false", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T03:40:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_inflight_ops": "1024", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_timeout": "10.000000", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "objectstore_blackhole": "false", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "openssl_engine_opts": "", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_max_ops": "4", 2024-09-16T03:40:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_slop": "0.020000", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_asio_thread_count": "2", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_async_recovery_min_cost": "100", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backfill_scan_max": "512", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backfill_scan_min": "64", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backoff_on_peering": "false", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T03:40:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_beacon_report_interval": "300", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_duration": "30", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_blkin_trace_all": "false", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T03:40:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_update_on_start": "true", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_message_cap": "256", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_op_priority": "63", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_watch_timeout": "30", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_command_max_records": "256", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_compact_on_start": "false", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_update_on_start": "true", 2024-09-16T03:40:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_op_order": "false", 2024-09-16T03:40:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_shutdown": "false", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_trim_objects": "false", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T03:40:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_default_notify_timeout": "30", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_enable_op_tracker": "true", 2024-09-16T03:40:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_info": "true", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_shutdown": "true", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_function_tracing": "false", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_grace": "20", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_interval": "6", 2024-09-16T03:40:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_stale": "600", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_journal_size": "5120", 2024-09-16T03:40:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_kill_backfill_at": "0", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_cache_size": "50", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_dedup": "true", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_message_max": "40", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_attr_name_len": "100", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_attr_size": "0", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_backfills": "1", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_markdown_count": "5", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_markdown_period": "600", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_object_name_len": "2048", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T03:40:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_object_size": "134217728", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pg_log_entries": "10000", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pgls": "1024", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_push_cost": "8388608", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_push_objects": "10", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_scrubs": "3", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T03:40:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_write_size": "90", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_profile": "balanced", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T03:40:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_base": "805306368", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_target": "4294967296", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_target_autotune": "false", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T03:40:07.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_min_pg_log_entries": "250", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_min_recovery_priority": "0", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_report_interval": "5", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_num_cache_shards": "32", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_numa_node": "-1", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objecter_finishers": "1", 2024-09-16T03:40:07.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objectstore": "bluestore", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objectstore_fuse": "false", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objectstore_tracing": "false", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_duration": "600", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_size": "20", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_log_threshold": "5", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_shards": "0", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T03:40:07.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_thread_timeout": "15", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_open_classes_on_start": "true", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_os_flags": "0", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_peering_op_priority": "255", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_delete_priority": "5", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T03:40:07.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_log_trim_min": "100", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_crimson": "false", 2024-09-16T03:40:07.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flags": "0", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_min_size": "0", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T03:40:07.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_read_ratio": "70", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_size": "2", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_type": "replicated", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_cost": "20971520", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T03:40:07.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_active": "0", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_op_priority": "3", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_priority": "5", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T03:40:07.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_repair_during_recovery": "false", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_chunk_max": "25", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_cost": "52428800", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T03:40:07.298 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_end_hour": "0", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_priority": "5", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-16T03:40:07.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_skip_data_digest": "false", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_smart_report_timeout": "5", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_priority": "5", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_target_transaction_size": "30", 2024-09-16T03:40:07.300 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tracing": "false", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_use_stale_snap": "false", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T03:40:07.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_kill_at": "0", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_max_join_drift": "10", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_min": "500", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_min_wait": "0.050000", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_service_trim_max": "500", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_service_trim_min": "250", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_stash_full_interval": "25", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_trim_max": "500", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_trim_min": "250", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "perf": "true", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-16T03:40:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "-", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrv": "", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "public_bind_addr": "-", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "public_network": "", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "public_network_interface": "", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "qat_compressor_busy_polling": "false", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "qat_compressor_enabled": "false", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "qat_compressor_session_max_number": "256", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rados_mon_op_timeout": "0", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rados_osd_op_timeout": "0", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rados_tracing": "false", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_atime_update_interval": "60", 2024-09-16T03:40:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache": "true", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_policy": "writearound", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_size": "33554432", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T03:40:07.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_compression_hint": "none", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_clone_format": "auto", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_data_pool": "", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_features": "61", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_format": "2", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_map_options": "", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_order": "22", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_pool": "rbd", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_stripe_count": "0", 2024-09-16T03:40:07.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_io_scheduler": "simple", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T03:40:07.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_order": "24", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_pool": "", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_splay_width": "4", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T03:40:07.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T03:40:07.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T03:40:07.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_op_threads": "1", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_plugins": "", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T03:40:07.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T03:40:07.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_tracing": "false", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_validate_names": "true", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_validate_pool": "true", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "restapi_base_url": "", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "restapi_log_level": "", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-16T03:40:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_admin_entry": "admin", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_backend_store": "rados", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_barbican_url": "", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_beast_enable_async": "true", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_counters_cache": "false", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T03:40:07.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cache_enabled": "true", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cache_lru_size": "25000", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_config_store": "rados", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_content_length_compat": "false", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cross_domain_policy": "", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T03:40:07.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T03:40:07.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T03:40:07.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T03:40:07.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d4n_port": "6379", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_window": "30", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-16T03:40:07.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T03:40:07.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dns_name": "", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dns_s3website_name": "", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T03:40:07.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_ops_log": "false", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_static_website": "false", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_usage_log": "false", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_expose_bucket": "false", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_extended_http_attrs": "", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_filter": "none", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T03:40:07.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_objs": "32", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_init_timeout": "300", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-16T03:40:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_password": "", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_project": "", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_token": "", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_user": "", 2024-09-16T03:40:07.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_api_version": "2", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_url": "", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T03:40:07.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_objs": "32", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_worker": "3", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_uri": "", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T03:40:07.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_http_headers": "", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_attr_size": "0", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-16T03:40:07.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_listing_results": "1000", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_notify_retries": "10", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T03:40:07.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T03:40:07.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_num_control_oids": "8", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_numa_node": "-1", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_op_tracing": "false", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_opa_token": "", 2024-09-16T03:40:07.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_opa_url": "", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_rados": "false", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-16T03:40:07.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_posix_cache_lanes": "3", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_posix_cache_partitions": "3", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_print_continue": "true", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rados_tracing": "false", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_realm": "", 2024-09-16T03:40:07.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_realm_id": "", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_region": "", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_request_uri": "", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T03:40:07.332 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_resolve_cname": "false", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_run_sync_thread": "true", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_script_uri": "", 2024-09-16T03:40:07.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_service_provider_name": "", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_client_id": "", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_client_secret": "", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_entry": "sts", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_key": "sts", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_auth_url": "", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_custom_header": "", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T03:40:07.334 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_need_stats": "true", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_tenant_name": "", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_url": "", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_lease_period": "120", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T03:40:07.335 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_thread_pool_size": "512", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_comment": "", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_createby": "", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_encoding": "", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_flag": "false", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-16T03:40:07.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_origin": "", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_tracker": "", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_max_shards": "32", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_use_opa_authz": "false", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_counters_cache": "false", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T03:40:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_policies_max_num": "100", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_unique_email": "true", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_verify_ssl": "true", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zone": "", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zone_id": "", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zonegroup": "", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zonegroup_id": "", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T03:40:07.338 INFO:teuthology.orchestra.run.smithi076.stdout: "rgwlc_auto_session_clear": "true", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_block_size": "4096", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T03:40:07.339 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_partition_filters": "false", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_perf": "false", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "run_dir": "/var/run/ceph", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_block_create": "true", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-16T03:40:07.340 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_device_size": "53687091200", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_full_integrity_check": "false", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_main_device_type": "SSD", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-16T03:40:07.341 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_segment_size": "67108864", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "set_keepcaps": "false", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "setgroup": "", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "setuser": "", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "setuser_match_path": "", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "thp": "false", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "threadpool_default_timeout": "60", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout: "throttler_perf_counter": "true" 2024-09-16T03:40:07.342 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:07.345 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T03:40:07.457 INFO:teuthology.orchestra.run.smithi076.stdout:mon.a: {} 2024-09-16T03:40:07.458 INFO:teuthology.orchestra.run.smithi076.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:40:07.469 INFO:teuthology.orchestra.run.smithi076.stdout:mon.b: {} 2024-09-16T03:40:07.469 INFO:teuthology.orchestra.run.smithi076.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:40:07.479 INFO:teuthology.orchestra.run.smithi076.stdout:mon.c: {} 2024-09-16T03:40:07.480 INFO:teuthology.orchestra.run.smithi076.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:40:07.490 DEBUG:teuthology.orchestra.run.smithi076:> type ceph-objectstore-tool 2024-09-16T03:40:07.504 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T03:40:07.505 DEBUG:teuthology.orchestra.run.smithi103:> type ceph-objectstore-tool 2024-09-16T03:40:07.522 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T03:40:07.522 DEBUG:teuthology.orchestra.run.smithi119:> type ceph-objectstore-tool 2024-09-16T03:40:07.540 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T03:40:07.541 DEBUG:teuthology.orchestra.run.smithi121:> type ceph-objectstore-tool 2024-09-16T03:40:07.561 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T03:40:07.562 INFO:teuthology.run_tasks:Running task rados... 2024-09-16T03:40:07.583 INFO:tasks.rados:Beginning rados... 2024-09-16T03:40:07.583 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-16T03:40:07.591 INFO:tasks.rados:joining rados 2024-09-16T03:40:07.591 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T03:40:07.591 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:40:07.591 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:40:07.592 INFO:tasks.thrashosds.thrasher:Reweighting osd 14 to 0.5100647073055062 2024-09-16T03:40:07.592 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 0.5100647073055062 2024-09-16T03:40:07.592 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T03:40:07.593 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T03:40:07.593 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:07.593 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T03:40:07.593 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:40:07.594 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T03:40:07.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:40:07.594 INFO:tasks.rados:clients are ['client.0'] 2024-09-16T03:40:07.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-09-16T03:40:07.603 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:07.601+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:07.694 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:07.692+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:07.750 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:07.750 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:07.750 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:07.750 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:07.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:40:07.791 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:07.792 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:07.795 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:07.793+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:07.824 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:07.859 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:07.868 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:07.868 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:07.877 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:40:07.894 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:07.894 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:07.896 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:07.894+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:07.915 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:07.915 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:07.940 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:07.940 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:07.957 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:07.957 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:07.974 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:07.974 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:07.992 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:07.993 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:07.997 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:07.995+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:08.009 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.009 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.009 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:08.010 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:08.018 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:40:08.026 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:08.027 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:08.044 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:08.044 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:08.060 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:08.061 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:08.078 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:08.078 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:08.095 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:08.094+0000 7fe4ae5e9640 0 -- 172.21.15.76:0/3895829861 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7fe484033250 msgr2=0x7fe484035640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:08.096 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:08.095+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:08.131 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.131 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.139 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:40:08.197 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:08.196+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:08.283 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.283 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.292 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:40:08.298 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:08.296+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:08.313 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:08.314 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.399 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:08.397+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:08.404 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:40:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.14 to 0.510065 (8293) 2024-09-16T03:40:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:load: shec 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T03:40:08.489 INFO:tasks.rados:starting run 0 out of 1 2024-09-16T03:40:08.489 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-16T03:40:08.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-09-16T03:40:08.532 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:08.531+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:08.555 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.555 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:40:08.600 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:08.599+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:08.701 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:08.700+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:08.711 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.712 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.721 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:40:08.803 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:08.801+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.875 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:40:08.919 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:08.917+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:40:09.005 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:09.003+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:09.105 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:09.104+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:09.132 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:09.132 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:09.141 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:40:09.206 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:09.205+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:09.284 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:09.285 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:09.293 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:40:09.307 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:09.306+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:09.408 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:09.406+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:09.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:40:09.510 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:09.507+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:09.564 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:09.564 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:09.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:40:09.613 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:09.611+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:09.712 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:09.711+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:09.715 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:09.715 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:09.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:40:09.813 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:09.812+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:09.834 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:40:09.914 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:09.912+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:09.938 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:09.938 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:09.946 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:40:10.015 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:10.014+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:10.090 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:10.090 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:10.099 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:40:10.116 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:10.115+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:10.202 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:10.202 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:10.210 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:40:10.217 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:10.216+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:10.318 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:10.316+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:10.323 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:40:10.325 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:10.419 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:10.417+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:10.451 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:10.451 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:10.468 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:10.468 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:10.468 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:10.469 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:10.477 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:40:10.485 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:10.486 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:10.486 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'unique_pool_0' created 2024-09-16T03:40:10.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:40:10.504 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-16T03:40:10.509 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:10.509 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:10.520 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:10.518+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:10.528 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:10.529 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:10.545 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:10.546 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:10.578 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:10.578 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:10.599 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:10.600 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:10.620 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:10.617+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:10.620 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:10.620 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:10.622 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:10.622 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:10.629 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:40:10.653 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:10.654 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:10.672 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:10.673 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:10.690 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:10.690 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:10.706 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:10.706 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:10.721 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:10.719+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:10.741 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:10.741 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:10.758 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:10.759 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:10.758+0000 7f282e8e8640 0 -- 172.21.15.76:0/4116283858 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f282821fc70 msgr2=0x7f2828222060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:10.773 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:10.773 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:10.773 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:10.773 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:10.773 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:10.777 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:10.777 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:10.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:40:10.821 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:10.820+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:10.884 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:10.885 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:10.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:10.885 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-09-16T03:40:10.885 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:10.893 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:40:10.938 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:10.936+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:11.008 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:11.008 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:11.009 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:11.009 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:11.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:40:11.059 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:11.057+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:11.124 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:11.122+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:11.161 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:11.161 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:11.161 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:11.161 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:11.161 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:11.170 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:40:11.224 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:11.222+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:11.273 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:11.273 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:11.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:11.273 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [] 2024-09-16T03:40:11.273 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:11.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:40:11.326 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:11.324+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:11.385 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:11.385 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:11.385 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:11.385 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:11.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:40:11.435 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:11.433+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:11.490 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T03:40:11.493 INFO:teuthology.orchestra.run.smithi076.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-16T03:40:11.547 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:11.547 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:11.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:40:11.557 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:11.555+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:11.627 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:11.626+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:11.659 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:11.659 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:11.659 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:11.659 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [] 2024-09-16T03:40:11.659 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:11.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:40:11.729 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:11.726+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:11.812 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:11.812 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:11.812 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:11.812 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:11.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:40:11.865 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:11.863+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:11.924 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:11.924 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:11.924 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:11.924 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:11.924 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:11.929 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:11.928+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:11.932 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:40:12.030 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:12.028+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:12.075 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.4341.0:5 2.2 e26/24 2:5cea7035:::rbd_info:head v 26'1, mlcod=0'0)", 2024-09-16T03:40:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:39:59.268638+0000", 2024-09-16T03:40:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 12.805392311, 2024-09-16T03:40:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00094064699999999997, 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:39:59.268638+0000", 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:39:59.268638+0000", 2024-09-16T03:40:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:39:59.268638+0000", 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.16e-07 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:39:59.268645+0000", 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6850000000000004e-06 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:39:59.268664+0000", 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8909000000000001e-05 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:39:59.268674+0000", 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0409e-05 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:39:59.268745+0000", 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0764000000000002e-05 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:39:59.268775+0000", 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0383999999999999e-05 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:39:59.269527+0000", 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00075221700000000001 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:39:59.269579+0000", 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1063000000000002e-05 2024-09-16T03:40:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.4341.0:7 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'2, mlcod=0'0)", 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:00.334230+0000", 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 11.739800038, 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0013741649999999999, 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:00.334230+0000", 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:00.334230+0000", 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:00.334231+0000", 2024-09-16T03:40:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5599999999999995e-07 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:00.334243+0000", 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.188e-05 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:00.334270+0000", 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7702999999999999e-05 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:00.334287+0000", 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7181999999999999e-05 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:00.334484+0000", 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000196863 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:00.334541+0000", 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7127999999999999e-05 2024-09-16T03:40:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:00.335566+0000", 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010244939999999999 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:00.335604+0000", 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8359000000000003e-05 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:12.085 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:12.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:40:12.170 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:12.168+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:12.229 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:12.229 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:12.232 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:12.230+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:12.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:40:12.333 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:12.331+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:12.381 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:12.382 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:12.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:40:12.433 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:12.432+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:12.493 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [] 2024-09-16T03:40:12.493 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:12.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:40:12.517 DEBUG:teuthology.orchestra.run.smithi121:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-09-16T03:40:12.547 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:12.545+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:12.576 INFO:tasks.rados.rados.0.smithi121.stdout:adding op weight read -> 100 2024-09-16T03:40:12.576 INFO:tasks.rados.rados.0.smithi121.stdout:adding op weight delete -> 50 2024-09-16T03:40:12.576 INFO:tasks.rados.rados.0.smithi121.stdout:adding op weight snap_create -> 50 2024-09-16T03:40:12.576 INFO:tasks.rados.rados.0.smithi121.stdout:adding op weight snap_remove -> 50 2024-09-16T03:40:12.576 INFO:tasks.rados.rados.0.smithi121.stdout:adding op weight rollback -> 50 2024-09-16T03:40:12.576 INFO:tasks.rados.rados.0.smithi121.stdout:adding op weight setattr -> 25 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout:adding op weight rmattr -> 25 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout:adding op weight copy_from -> 50 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout:adding op weight append -> 100 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout:ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout:Configuration: 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout: Number of operations: 400 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout: Number of objects: 50 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout: Max in flight operations: 16 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout: Object size (in bytes): 4000000 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout: Write stride min: 400000 2024-09-16T03:40:12.577 INFO:tasks.rados.rados.0.smithi121.stdout: Write stride max: 800000 2024-09-16T03:40:12.621 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:12.621 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:12.621 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:12.621 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:12.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:40:12.635 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:12.633+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:12.736 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:12.734+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:12.774 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:12.774 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:12.774 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:12.774 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:12.774 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:12.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:40:12.791 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:12.837 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:12.834+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:12.885 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [] 2024-09-16T03:40:12.885 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:12.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:40:12.931 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:12.931 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:12.937 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:12.935+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:12.949 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:12.949 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:12.965 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:12.966 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:12.982 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:12.982 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:12.997 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:12.997 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:12.997 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:12.997 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:12.997 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:13.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:40:13.014 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:13.015 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:13.031 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:13.040 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:13.036+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:13.048 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:13.049 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:13.064 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:13.081 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:13.098 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:13.098 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:13.110 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:13.110 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:13.115 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:13.115 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:13.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:40:13.130 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:13.130 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:13.147 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:13.161 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:13.160+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:13.163 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:13.161+0000 7f63d37fe640 0 -- 172.21.15.76:0/4148647082 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f63d4223640 msgr2=0x7f63d4225a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:13.222 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:13.222 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:13.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:40:13.272 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:13.270+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:13.342 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:13.340+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:13.376 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:13.376 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:13.381 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:13.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:40:13.468 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:13.466+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:13.489 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:40:13.489 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:40:13.489 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T03:40:13.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T03:40:13.510 INFO:tasks.rados.rados.0.smithi121.stdout:1: write initial oid 1 2024-09-16T03:40:13.510 INFO:tasks.rados.rados.0.smithi121.stdout:1: seq_num 0 ranges {0=32768} 2024-09-16T03:40:13.510 INFO:tasks.rados.rados.0.smithi121.stdout:1: writing smithi12138290-1 from 0 to 32768 tid 1 2024-09-16T03:40:13.510 INFO:tasks.rados.rados.0.smithi121.stdout:2: write initial oid 2 2024-09-16T03:40:13.510 INFO:tasks.rados.rados.0.smithi121.stdout:2: seq_num 1 ranges {0=32768} 2024-09-16T03:40:13.510 INFO:tasks.rados.rados.0.smithi121.stdout:2: writing smithi12138290-2 from 0 to 32768 tid 1 2024-09-16T03:40:13.510 INFO:tasks.rados.rados.0.smithi121.stdout:3: write initial oid 3 2024-09-16T03:40:13.510 INFO:tasks.rados.rados.0.smithi121.stdout:3: seq_num 2 ranges {0=49152} 2024-09-16T03:40:13.510 INFO:tasks.rados.rados.0.smithi121.stdout:3: writing smithi12138290-3 from 0 to 49152 tid 1 2024-09-16T03:40:13.511 INFO:tasks.rados.rados.0.smithi121.stdout:4: write initial oid 4 2024-09-16T03:40:13.511 INFO:tasks.rados.rados.0.smithi121.stdout:4: seq_num 3 ranges {0=65536} 2024-09-16T03:40:13.511 INFO:tasks.rados.rados.0.smithi121.stdout:4: writing smithi12138290-4 from 0 to 65536 tid 1 2024-09-16T03:40:13.511 INFO:tasks.rados.rados.0.smithi121.stdout:5: write initial oid 5 2024-09-16T03:40:13.512 INFO:tasks.rados.rados.0.smithi121.stdout:5: seq_num 4 ranges {0=81920} 2024-09-16T03:40:13.512 INFO:tasks.rados.rados.0.smithi121.stdout:5: writing smithi12138290-5 from 0 to 81920 tid 1 2024-09-16T03:40:13.512 INFO:tasks.rados.rados.0.smithi121.stdout:6: write initial oid 6 2024-09-16T03:40:13.512 INFO:tasks.rados.rados.0.smithi121.stdout:6: seq_num 5 ranges {0=98304} 2024-09-16T03:40:13.512 INFO:tasks.rados.rados.0.smithi121.stdout:6: writing smithi12138290-6 from 0 to 98304 tid 1 2024-09-16T03:40:13.512 INFO:tasks.rados.rados.0.smithi121.stdout:7: write initial oid 7 2024-09-16T03:40:13.512 INFO:tasks.rados.rados.0.smithi121.stdout:7: seq_num 6 ranges {0=114688} 2024-09-16T03:40:13.513 INFO:tasks.rados.rados.0.smithi121.stdout:7: writing smithi12138290-7 from 0 to 114688 tid 1 2024-09-16T03:40:13.513 INFO:tasks.rados.rados.0.smithi121.stdout:8: write initial oid 8 2024-09-16T03:40:13.513 INFO:tasks.rados.rados.0.smithi121.stdout:8: seq_num 7 ranges {0=131072} 2024-09-16T03:40:13.514 INFO:tasks.rados.rados.0.smithi121.stdout:8: writing smithi12138290-8 from 0 to 131072 tid 1 2024-09-16T03:40:13.514 INFO:tasks.rados.rados.0.smithi121.stdout:9: write initial oid 9 2024-09-16T03:40:13.514 INFO:tasks.rados.rados.0.smithi121.stdout:9: seq_num 8 ranges {0=147456} 2024-09-16T03:40:13.514 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:40:13.517 INFO:tasks.rados.rados.0.smithi121.stdout:9: writing smithi12138290-9 from 0 to 147456 tid 1 2024-09-16T03:40:13.517 INFO:tasks.rados.rados.0.smithi121.stdout:10: write initial oid 10 2024-09-16T03:40:13.517 INFO:tasks.rados.rados.0.smithi121.stdout:10: seq_num 9 ranges {0=163840} 2024-09-16T03:40:13.517 INFO:tasks.rados.rados.0.smithi121.stdout:10: writing smithi12138290-10 from 0 to 163840 tid 1 2024-09-16T03:40:13.518 INFO:tasks.rados.rados.0.smithi121.stdout:11: write initial oid 11 2024-09-16T03:40:13.518 INFO:tasks.rados.rados.0.smithi121.stdout:11: seq_num 10 ranges {0=180224} 2024-09-16T03:40:13.518 INFO:tasks.rados.rados.0.smithi121.stdout:11: writing smithi12138290-11 from 0 to 180224 tid 1 2024-09-16T03:40:13.518 INFO:tasks.rados.rados.0.smithi121.stdout:1: finishing write tid 1 to smithi12138290-1 2024-09-16T03:40:13.519 INFO:tasks.rados.rados.0.smithi121.stdout:2: finishing write tid 1 to smithi12138290-2 2024-09-16T03:40:13.519 INFO:tasks.rados.rados.0.smithi121.stdout:1: finishing write tid 2 to smithi12138290-1 2024-09-16T03:40:13.519 INFO:tasks.rados.rados.0.smithi121.stdout:12: write initial oid 12 2024-09-16T03:40:13.519 INFO:tasks.rados.rados.0.smithi121.stdout:12: seq_num 11 ranges {0=196608} 2024-09-16T03:40:13.520 INFO:tasks.rados.rados.0.smithi121.stdout:12: writing smithi12138290-12 from 0 to 196608 tid 1 2024-09-16T03:40:13.520 INFO:tasks.rados.rados.0.smithi121.stdout:13: write initial oid 13 2024-09-16T03:40:13.520 INFO:tasks.rados.rados.0.smithi121.stdout:13: seq_num 12 ranges {0=212992} 2024-09-16T03:40:13.522 INFO:tasks.rados.rados.0.smithi121.stdout:13: writing smithi12138290-13 from 0 to 212992 tid 1 2024-09-16T03:40:13.522 INFO:tasks.rados.rados.0.smithi121.stdout:14: write initial oid 14 2024-09-16T03:40:13.522 INFO:tasks.rados.rados.0.smithi121.stdout:14: seq_num 13 ranges {0=229376} 2024-09-16T03:40:13.523 INFO:tasks.rados.rados.0.smithi121.stdout:14: writing smithi12138290-14 from 0 to 229376 tid 1 2024-09-16T03:40:13.523 INFO:tasks.rados.rados.0.smithi121.stdout:15: write initial oid 15 2024-09-16T03:40:13.523 INFO:tasks.rados.rados.0.smithi121.stdout:15: seq_num 14 ranges {0=245760} 2024-09-16T03:40:13.525 INFO:tasks.rados.rados.0.smithi121.stdout:15: writing smithi12138290-15 from 0 to 245760 tid 1 2024-09-16T03:40:13.525 INFO:tasks.rados.rados.0.smithi121.stdout:16: write initial oid 16 2024-09-16T03:40:13.526 INFO:tasks.rados.rados.0.smithi121.stdout:16: seq_num 15 ranges {0=262144} 2024-09-16T03:40:13.528 INFO:tasks.rados.rados.0.smithi121.stdout:16: writing smithi12138290-16 from 0 to 262144 tid 1 2024-09-16T03:40:13.528 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.528 INFO:tasks.rados.rados.0.smithi121.stdout:3: finishing write tid 1 to smithi12138290-3 2024-09-16T03:40:13.528 INFO:tasks.rados.rados.0.smithi121.stdout:4: finishing write tid 1 to smithi12138290-4 2024-09-16T03:40:13.528 INFO:tasks.rados.rados.0.smithi121.stdout:2: finishing write tid 2 to smithi12138290-2 2024-09-16T03:40:13.528 INFO:tasks.rados.rados.0.smithi121.stdout:3: finishing write tid 2 to smithi12138290-3 2024-09-16T03:40:13.528 INFO:tasks.rados.rados.0.smithi121.stdout:4: finishing write tid 2 to smithi12138290-4 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:2: finishing write tid 3 to smithi12138290-2 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:2: oid 2 updating version 0 to 1 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:2: oid 2 updating version 1 to 2 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:3: finishing write tid 3 to smithi12138290-3 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:3: oid 3 updating version 0 to 1 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:3: oid 3 updating version 1 to 2 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:6: finishing write tid 1 to smithi12138290-6 2024-09-16T03:40:13.529 INFO:tasks.rados.rados.0.smithi121.stdout:7: finishing write tid 1 to smithi12138290-7 2024-09-16T03:40:13.530 INFO:tasks.rados.rados.0.smithi121.stdout:6: finishing write tid 2 to smithi12138290-6 2024-09-16T03:40:13.530 INFO:tasks.rados.rados.0.smithi121.stdout:7: finishing write tid 2 to smithi12138290-7 2024-09-16T03:40:13.530 INFO:tasks.rados.rados.0.smithi121.stdout:2: done (15 left) 2024-09-16T03:40:13.530 INFO:tasks.rados.rados.0.smithi121.stdout:3: done (14 left) 2024-09-16T03:40:13.530 INFO:tasks.rados.rados.0.smithi121.stdout:17: write initial oid 17 2024-09-16T03:40:13.530 INFO:tasks.rados.rados.0.smithi121.stdout:17: seq_num 16 ranges {0=278528} 2024-09-16T03:40:13.531 INFO:tasks.rados.rados.0.smithi121.stdout:17: writing smithi12138290-17 from 0 to 278528 tid 1 2024-09-16T03:40:13.531 INFO:tasks.rados.rados.0.smithi121.stdout:18: write initial oid 18 2024-09-16T03:40:13.531 INFO:tasks.rados.rados.0.smithi121.stdout:18: seq_num 17 ranges {0=294912} 2024-09-16T03:40:13.531 INFO:tasks.rados.rados.0.smithi121.stdout:18: writing smithi12138290-18 from 0 to 294912 tid 1 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:5: finishing write tid 1 to smithi12138290-5 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:5: finishing write tid 2 to smithi12138290-5 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:10: finishing write tid 1 to smithi12138290-10 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:6: finishing write tid 3 to smithi12138290-6 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:6: oid 6 updating version 0 to 1 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:6: oid 6 updating version 1 to 2 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:10: finishing write tid 2 to smithi12138290-10 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:6: done (15 left) 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:19: write initial oid 19 2024-09-16T03:40:13.532 INFO:tasks.rados.rados.0.smithi121.stdout:19: seq_num 18 ranges {0=311296} 2024-09-16T03:40:13.533 INFO:tasks.rados.rados.0.smithi121.stdout:19: writing smithi12138290-19 from 0 to 311296 tid 1 2024-09-16T03:40:13.533 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.533 INFO:tasks.rados.rados.0.smithi121.stdout:9: finishing write tid 1 to smithi12138290-9 2024-09-16T03:40:13.533 INFO:tasks.rados.rados.0.smithi121.stdout:9: finishing write tid 2 to smithi12138290-9 2024-09-16T03:40:13.536 INFO:tasks.rados.rados.0.smithi121.stdout:9: finishing write tid 3 to smithi12138290-9 2024-09-16T03:40:13.536 INFO:tasks.rados.rados.0.smithi121.stdout:9: oid 9 updating version 0 to 2 2024-09-16T03:40:13.536 INFO:tasks.rados.rados.0.smithi121.stdout:9: oid 9 version 2 is already newer than 1 2024-09-16T03:40:13.536 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:40:13.536 INFO:tasks.rados.rados.0.smithi121.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-16T03:40:13.536 INFO:tasks.rados.rados.0.smithi121.stdout:9: done (15 left) 2024-09-16T03:40:13.536 INFO:tasks.rados.rados.0.smithi121.stdout:20: write initial oid 20 2024-09-16T03:40:13.537 INFO:tasks.rados.rados.0.smithi121.stdout:20: seq_num 19 ranges {0=327680} 2024-09-16T03:40:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:13.538 INFO:tasks.rados.rados.0.smithi121.stdout:20: writing smithi12138290-20 from 0 to 327680 tid 1 2024-09-16T03:40:13.538 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.538 INFO:tasks.rados.rados.0.smithi121.stdout:17: finishing write tid 1 to smithi12138290-17 2024-09-16T03:40:13.538 INFO:tasks.rados.rados.0.smithi121.stdout:16: finishing write tid 1 to smithi12138290-16 2024-09-16T03:40:13.538 INFO:tasks.rados.rados.0.smithi121.stdout:16: finishing write tid 2 to smithi12138290-16 2024-09-16T03:40:13.538 INFO:tasks.rados.rados.0.smithi121.stdout:17: finishing write tid 2 to smithi12138290-17 2024-09-16T03:40:13.538 INFO:tasks.rados.rados.0.smithi121.stdout:10: finishing write tid 3 to smithi12138290-10 2024-09-16T03:40:13.539 INFO:tasks.rados.rados.0.smithi121.stdout:10: oid 10 updating version 0 to 1 2024-09-16T03:40:13.539 INFO:tasks.rados.rados.0.smithi121.stdout:10: oid 10 updating version 1 to 2 2024-09-16T03:40:13.539 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:40:13.539 INFO:tasks.rados.rados.0.smithi121.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:40:13.539 INFO:tasks.rados.rados.0.smithi121.stdout:10: done (15 left) 2024-09-16T03:40:13.539 INFO:tasks.rados.rados.0.smithi121.stdout:21: write initial oid 21 2024-09-16T03:40:13.539 INFO:tasks.rados.rados.0.smithi121.stdout:21: seq_num 20 ranges {0=344064} 2024-09-16T03:40:13.540 INFO:tasks.rados.rados.0.smithi121.stdout:21: writing smithi12138290-21 from 0 to 344064 tid 1 2024-09-16T03:40:13.540 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.540 INFO:tasks.rados.rados.0.smithi121.stdout:17: finishing write tid 3 to smithi12138290-17 2024-09-16T03:40:13.540 INFO:tasks.rados.rados.0.smithi121.stdout:17: oid 17 updating version 0 to 3 2024-09-16T03:40:13.541 INFO:tasks.rados.rados.0.smithi121.stdout:17: oid 17 updating version 3 to 4 2024-09-16T03:40:13.541 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 17 v 4 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:40:13.541 INFO:tasks.rados.rados.0.smithi121.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-16T03:40:13.541 INFO:tasks.rados.rados.0.smithi121.stdout:17: done (15 left) 2024-09-16T03:40:13.541 INFO:tasks.rados.rados.0.smithi121.stdout:22: write initial oid 22 2024-09-16T03:40:13.541 INFO:tasks.rados.rados.0.smithi121.stdout:22: seq_num 21 ranges {0=360448} 2024-09-16T03:40:13.544 INFO:tasks.rados.rados.0.smithi121.stdout:22: writing smithi12138290-22 from 0 to 360448 tid 1 2024-09-16T03:40:13.544 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.544 INFO:tasks.rados.rados.0.smithi121.stdout:4: finishing write tid 3 to smithi12138290-4 2024-09-16T03:40:13.544 INFO:tasks.rados.rados.0.smithi121.stdout:4: oid 4 updating version 0 to 1 2024-09-16T03:40:13.544 INFO:tasks.rados.rados.0.smithi121.stdout:4: oid 4 updating version 1 to 2 2024-09-16T03:40:13.544 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:40:13.544 INFO:tasks.rados.rados.0.smithi121.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-16T03:40:13.544 INFO:tasks.rados.rados.0.smithi121.stdout:16: finishing write tid 3 to smithi12138290-16 2024-09-16T03:40:13.544 INFO:tasks.rados.rados.0.smithi121.stdout:16: oid 16 updating version 0 to 4 2024-09-16T03:40:13.544 INFO:tasks.rados.rados.0.smithi121.stdout:16: oid 16 version 4 is already newer than 3 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 16 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:1: finishing write tid 3 to smithi12138290-1 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:1: oid 1 updating version 0 to 1 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:1: oid 1 updating version 1 to 2 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:1: done (15 left) 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:4: done (14 left) 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:16: done (13 left) 2024-09-16T03:40:13.545 INFO:tasks.rados.rados.0.smithi121.stdout:23: write initial oid 23 2024-09-16T03:40:13.546 INFO:tasks.rados.rados.0.smithi121.stdout:23: seq_num 22 ranges {0=376832} 2024-09-16T03:40:13.547 INFO:tasks.rados.rados.0.smithi121.stdout:23: writing smithi12138290-23 from 0 to 376832 tid 1 2024-09-16T03:40:13.547 INFO:tasks.rados.rados.0.smithi121.stdout:24: write initial oid 24 2024-09-16T03:40:13.547 INFO:tasks.rados.rados.0.smithi121.stdout:24: seq_num 23 ranges {0=393216} 2024-09-16T03:40:13.547 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:13.545+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:13.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:40:13.549 INFO:tasks.rados.rados.0.smithi121.stdout:24: writing smithi12138290-24 from 0 to 393216 tid 1 2024-09-16T03:40:13.549 INFO:tasks.rados.rados.0.smithi121.stdout:25: write initial oid 25 2024-09-16T03:40:13.549 INFO:tasks.rados.rados.0.smithi121.stdout:25: seq_num 24 ranges {0=409600} 2024-09-16T03:40:13.550 INFO:tasks.rados.rados.0.smithi121.stdout:25: writing smithi12138290-25 from 0 to 409600 tid 1 2024-09-16T03:40:13.550 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:13: finishing write tid 1 to smithi12138290-13 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:13: finishing write tid 2 to smithi12138290-13 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:13: finishing write tid 3 to smithi12138290-13 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:13: oid 13 updating version 0 to 2 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:13: oid 13 version 2 is already newer than 1 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 13 v 2 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:12: finishing write tid 1 to smithi12138290-12 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:12: finishing write tid 2 to smithi12138290-12 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:14: finishing write tid 1 to smithi12138290-14 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:14: finishing write tid 2 to smithi12138290-14 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:15: finishing write tid 1 to smithi12138290-15 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:15: finishing write tid 2 to smithi12138290-15 2024-09-16T03:40:13.551 INFO:tasks.rados.rados.0.smithi121.stdout:8: finishing write tid 1 to smithi12138290-8 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:8: finishing write tid 2 to smithi12138290-8 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:11: finishing write tid 1 to smithi12138290-11 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:11: finishing write tid 2 to smithi12138290-11 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:14: finishing write tid 3 to smithi12138290-14 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:14: oid 14 updating version 0 to 4 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:14: oid 14 version 4 is already newer than 3 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 14 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:20: finishing write tid 1 to smithi12138290-20 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:20: finishing write tid 2 to smithi12138290-20 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:5: finishing write tid 3 to smithi12138290-5 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:5: oid 5 updating version 0 to 1 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:5: oid 5 updating version 1 to 2 2024-09-16T03:40:13.552 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:40:13.553 INFO:tasks.rados.rados.0.smithi121.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-16T03:40:13.553 INFO:tasks.rados.rados.0.smithi121.stdout:5: done (15 left) 2024-09-16T03:40:13.553 INFO:tasks.rados.rados.0.smithi121.stdout:13: done (14 left) 2024-09-16T03:40:13.553 INFO:tasks.rados.rados.0.smithi121.stdout:14: done (13 left) 2024-09-16T03:40:13.553 INFO:tasks.rados.rados.0.smithi121.stdout:26: write initial oid 26 2024-09-16T03:40:13.553 INFO:tasks.rados.rados.0.smithi121.stdout:26: seq_num 25 ranges {0=425984} 2024-09-16T03:40:13.554 INFO:tasks.rados.rados.0.smithi121.stdout:26: writing smithi12138290-26 from 0 to 425984 tid 1 2024-09-16T03:40:13.554 INFO:tasks.rados.rados.0.smithi121.stdout:27: write initial oid 27 2024-09-16T03:40:13.554 INFO:tasks.rados.rados.0.smithi121.stdout:27: seq_num 26 ranges {0=442368} 2024-09-16T03:40:13.556 INFO:tasks.rados.rados.0.smithi121.stdout:27: writing smithi12138290-27 from 0 to 442368 tid 1 2024-09-16T03:40:13.556 INFO:tasks.rados.rados.0.smithi121.stdout:28: write initial oid 28 2024-09-16T03:40:13.556 INFO:tasks.rados.rados.0.smithi121.stdout:28: seq_num 27 ranges {0=458752} 2024-09-16T03:40:13.559 INFO:tasks.rados.rados.0.smithi121.stdout:28: writing smithi12138290-28 from 0 to 458752 tid 1 2024-09-16T03:40:13.559 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.559 INFO:tasks.rados.rados.0.smithi121.stdout:22: finishing write tid 1 to smithi12138290-22 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:22: finishing write tid 2 to smithi12138290-22 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:12: finishing write tid 3 to smithi12138290-12 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:12: oid 12 updating version 0 to 1 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:12: oid 12 updating version 1 to 2 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:20: finishing write tid 3 to smithi12138290-20 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:20: oid 20 updating version 0 to 4 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:20: oid 20 version 4 is already newer than 3 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 20 v 4 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:21: finishing write tid 1 to smithi12138290-21 2024-09-16T03:40:13.560 INFO:tasks.rados.rados.0.smithi121.stdout:15: finishing write tid 3 to smithi12138290-15 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:15: oid 15 updating version 0 to 3 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:15: oid 15 updating version 3 to 4 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:22: finishing write tid 3 to smithi12138290-22 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:22: oid 22 updating version 0 to 6 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:22: oid 22 version 6 is already newer than 5 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:21: finishing write tid 2 to smithi12138290-21 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:7: finishing write tid 3 to smithi12138290-7 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:7: oid 7 updating version 0 to 1 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:7: oid 7 updating version 1 to 2 2024-09-16T03:40:13.561 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:40:13.562 INFO:tasks.rados.rados.0.smithi121.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-16T03:40:13.562 INFO:tasks.rados.rados.0.smithi121.stdout:18: finishing write tid 1 to smithi12138290-18 2024-09-16T03:40:13.562 INFO:tasks.rados.rados.0.smithi121.stdout:7: done (15 left) 2024-09-16T03:40:13.562 INFO:tasks.rados.rados.0.smithi121.stdout:12: done (14 left) 2024-09-16T03:40:13.562 INFO:tasks.rados.rados.0.smithi121.stdout:15: done (13 left) 2024-09-16T03:40:13.562 INFO:tasks.rados.rados.0.smithi121.stdout:20: done (12 left) 2024-09-16T03:40:13.562 INFO:tasks.rados.rados.0.smithi121.stdout:22: done (11 left) 2024-09-16T03:40:13.562 INFO:tasks.rados.rados.0.smithi121.stdout:29: write initial oid 29 2024-09-16T03:40:13.562 INFO:tasks.rados.rados.0.smithi121.stdout:29: seq_num 28 ranges {0=475136} 2024-09-16T03:40:13.563 INFO:tasks.rados.rados.0.smithi121.stdout:29: writing smithi12138290-29 from 0 to 475136 tid 1 2024-09-16T03:40:13.563 INFO:tasks.rados.rados.0.smithi121.stdout:30: write initial oid 30 2024-09-16T03:40:13.563 INFO:tasks.rados.rados.0.smithi121.stdout:30: seq_num 29 ranges {0=491520} 2024-09-16T03:40:13.565 INFO:tasks.rados.rados.0.smithi121.stdout:30: writing smithi12138290-30 from 0 to 491520 tid 1 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:31: write initial oid 31 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:24: finishing write tid 1 to smithi12138290-24 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:24: finishing write tid 2 to smithi12138290-24 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:18: finishing write tid 2 to smithi12138290-18 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:8: finishing write tid 3 to smithi12138290-8 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:8: oid 8 updating version 0 to 3 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:8: oid 8 updating version 3 to 4 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 8 v 4 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:25: finishing write tid 1 to smithi12138290-25 2024-09-16T03:40:13.566 INFO:tasks.rados.rados.0.smithi121.stdout:31: seq_num 30 ranges {0=507904} 2024-09-16T03:40:13.569 INFO:tasks.rados.rados.0.smithi121.stdout:31: writing smithi12138290-31 from 0 to 507904 tid 1 2024-09-16T03:40:13.569 INFO:tasks.rados.rados.0.smithi121.stdout:8: done (13 left) 2024-09-16T03:40:13.569 INFO:tasks.rados.rados.0.smithi121.stdout:32: write initial oid 32 2024-09-16T03:40:13.569 INFO:tasks.rados.rados.0.smithi121.stdout:32: seq_num 31 ranges {0=524288} 2024-09-16T03:40:13.572 INFO:tasks.rados.rados.0.smithi121.stdout:32: writing smithi12138290-32 from 0 to 524288 tid 1 2024-09-16T03:40:13.572 INFO:tasks.rados.rados.0.smithi121.stdout:33: write initial oid 33 2024-09-16T03:40:13.572 INFO:tasks.rados.rados.0.smithi121.stdout:33: seq_num 32 ranges {0=540672} 2024-09-16T03:40:13.575 INFO:tasks.rados.rados.0.smithi121.stdout:33: writing smithi12138290-33 from 0 to 540672 tid 1 2024-09-16T03:40:13.575 INFO:tasks.rados.rados.0.smithi121.stdout:34: write initial oid 34 2024-09-16T03:40:13.575 INFO:tasks.rados.rados.0.smithi121.stdout:34: seq_num 33 ranges {0=557056} 2024-09-16T03:40:13.579 INFO:tasks.rados.rados.0.smithi121.stdout:34: writing smithi12138290-34 from 0 to 557056 tid 1 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:11: finishing write tid 3 to smithi12138290-11 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:11: oid 11 updating version 0 to 6 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:11: oid 11 version 6 is already newer than 5 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:25: finishing write tid 2 to smithi12138290-25 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:19: finishing write tid 1 to smithi12138290-19 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:19: finishing write tid 2 to smithi12138290-19 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:23: finishing write tid 1 to smithi12138290-23 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:21: finishing write tid 3 to smithi12138290-21 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:21: oid 21 updating version 0 to 3 2024-09-16T03:40:13.580 INFO:tasks.rados.rados.0.smithi121.stdout:21: oid 21 updating version 3 to 4 2024-09-16T03:40:13.581 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 21 v 4 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:40:13.581 INFO:tasks.rados.rados.0.smithi121.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-16T03:40:13.581 INFO:tasks.rados.rados.0.smithi121.stdout:26: finishing write tid 1 to smithi12138290-26 2024-09-16T03:40:13.581 INFO:tasks.rados.rados.0.smithi121.stdout:11: done (15 left) 2024-09-16T03:40:13.581 INFO:tasks.rados.rados.0.smithi121.stdout:21: done (14 left) 2024-09-16T03:40:13.581 INFO:tasks.rados.rados.0.smithi121.stdout:35: write initial oid 35 2024-09-16T03:40:13.581 INFO:tasks.rados.rados.0.smithi121.stdout:35: seq_num 34 ranges {0=573440} 2024-09-16T03:40:13.582 INFO:tasks.rados.rados.0.smithi121.stdout:35: writing smithi12138290-35 from 0 to 573440 tid 1 2024-09-16T03:40:13.582 INFO:tasks.rados.rados.0.smithi121.stdout:36: write initial oid 36 2024-09-16T03:40:13.582 INFO:tasks.rados.rados.0.smithi121.stdout:36: seq_num 35 ranges {0=589824} 2024-09-16T03:40:13.587 INFO:tasks.rados.rados.0.smithi121.stdout:36: writing smithi12138290-36 from 0 to 589824 tid 1 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:26: finishing write tid 2 to smithi12138290-26 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:23: finishing write tid 2 to smithi12138290-23 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:24: finishing write tid 3 to smithi12138290-24 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:24: oid 24 updating version 0 to 4 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:24: oid 24 version 4 is already newer than 3 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:25: finishing write tid 3 to smithi12138290-25 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:25: oid 25 updating version 0 to 5 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:25: oid 25 updating version 5 to 6 2024-09-16T03:40:13.588 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:18: finishing write tid 3 to smithi12138290-18 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:18: oid 18 updating version 0 to 8 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:18: oid 18 version 8 is already newer than 7 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 18 v 8 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:19: finishing write tid 3 to smithi12138290-19 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:19: oid 19 updating version 0 to 9 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:19: oid 19 updating version 9 to 10 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 19 v 10 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:26: finishing write tid 3 to smithi12138290-26 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:26: oid 26 updating version 0 to 6 2024-09-16T03:40:13.589 INFO:tasks.rados.rados.0.smithi121.stdout:26: oid 26 version 6 is already newer than 5 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:23: finishing write tid 3 to smithi12138290-23 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:23: oid 23 updating version 0 to 11 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:23: oid 23 updating version 11 to 12 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 23 v 12 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:28: finishing write tid 1 to smithi12138290-28 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:18: done (15 left) 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:19: done (14 left) 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:23: done (13 left) 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:24: done (12 left) 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:25: done (11 left) 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:26: done (10 left) 2024-09-16T03:40:13.590 INFO:tasks.rados.rados.0.smithi121.stdout:37: write initial oid 37 2024-09-16T03:40:13.591 INFO:tasks.rados.rados.0.smithi121.stdout:37: seq_num 36 ranges {0=606208} 2024-09-16T03:40:13.591 INFO:tasks.rados.rados.0.smithi121.stdout:37: writing smithi12138290-37 from 0 to 606208 tid 1 2024-09-16T03:40:13.591 INFO:tasks.rados.rados.0.smithi121.stdout:38: write initial oid 38 2024-09-16T03:40:13.591 INFO:tasks.rados.rados.0.smithi121.stdout:38: seq_num 37 ranges {0=622592} 2024-09-16T03:40:13.596 INFO:tasks.rados.rados.0.smithi121.stdout:38: writing smithi12138290-38 from 0 to 622592 tid 1 2024-09-16T03:40:13.596 INFO:tasks.rados.rados.0.smithi121.stdout:39: write initial oid 39 2024-09-16T03:40:13.596 INFO:tasks.rados.rados.0.smithi121.stdout:39: seq_num 38 ranges {0=638976} 2024-09-16T03:40:13.600 INFO:tasks.rados.rados.0.smithi121.stdout:39: writing smithi12138290-39 from 0 to 638976 tid 1 2024-09-16T03:40:13.600 INFO:tasks.rados.rados.0.smithi121.stdout:40: write initial oid 40 2024-09-16T03:40:13.601 INFO:tasks.rados.rados.0.smithi121.stdout:40: seq_num 39 ranges {0=671744} 2024-09-16T03:40:13.606 INFO:tasks.rados.rados.0.smithi121.stdout:40: writing smithi12138290-40 from 0 to 671744 tid 1 2024-09-16T03:40:13.606 INFO:tasks.rados.rados.0.smithi121.stdout:41: write initial oid 41 2024-09-16T03:40:13.606 INFO:tasks.rados.rados.0.smithi121.stdout:41: seq_num 40 ranges {0=688128} 2024-09-16T03:40:13.610 INFO:tasks.rados.rados.0.smithi121.stdout:41: writing smithi12138290-41 from 0 to 688128 tid 1 2024-09-16T03:40:13.611 INFO:tasks.rados.rados.0.smithi121.stdout:42: write initial oid 42 2024-09-16T03:40:13.611 INFO:tasks.rados.rados.0.smithi121.stdout:42: seq_num 41 ranges {0=704512} 2024-09-16T03:40:13.617 INFO:tasks.rados.rados.0.smithi121.stdout:42: writing smithi12138290-42 from 0 to 704512 tid 1 2024-09-16T03:40:13.617 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.617 INFO:tasks.rados.rados.0.smithi121.stdout:28: finishing write tid 2 to smithi12138290-28 2024-09-16T03:40:13.617 INFO:tasks.rados.rados.0.smithi121.stdout:29: finishing write tid 1 to smithi12138290-29 2024-09-16T03:40:13.617 INFO:tasks.rados.rados.0.smithi121.stdout:29: finishing write tid 2 to smithi12138290-29 2024-09-16T03:40:13.617 INFO:tasks.rados.rados.0.smithi121.stdout:28: finishing write tid 3 to smithi12138290-28 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:28: oid 28 updating version 0 to 2 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:28: oid 28 version 2 is already newer than 1 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 28 v 2 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:29: finishing write tid 3 to smithi12138290-29 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:29: oid 29 updating version 0 to 3 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:29: oid 29 updating version 3 to 4 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 29 v 4 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:30: finishing write tid 1 to smithi12138290-30 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:31: finishing write tid 1 to smithi12138290-31 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:30: finishing write tid 2 to smithi12138290-30 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:31: finishing write tid 2 to smithi12138290-31 2024-09-16T03:40:13.618 INFO:tasks.rados.rados.0.smithi121.stdout:31: finishing write tid 3 to smithi12138290-31 2024-09-16T03:40:13.619 INFO:tasks.rados.rados.0.smithi121.stdout:31: oid 31 updating version 0 to 4 2024-09-16T03:40:13.619 INFO:tasks.rados.rados.0.smithi121.stdout:31: oid 31 version 4 is already newer than 3 2024-09-16T03:40:13.619 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 31 v 4 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:40:13.619 INFO:tasks.rados.rados.0.smithi121.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-16T03:40:13.619 INFO:tasks.rados.rados.0.smithi121.stdout:28: done (15 left) 2024-09-16T03:40:13.619 INFO:tasks.rados.rados.0.smithi121.stdout:29: done (14 left) 2024-09-16T03:40:13.619 INFO:tasks.rados.rados.0.smithi121.stdout:31: done (13 left) 2024-09-16T03:40:13.619 INFO:tasks.rados.rados.0.smithi121.stdout:43: write initial oid 43 2024-09-16T03:40:13.619 INFO:tasks.rados.rados.0.smithi121.stdout:43: seq_num 42 ranges {0=720896} 2024-09-16T03:40:13.620 INFO:tasks.rados.rados.0.smithi121.stdout:43: writing smithi12138290-43 from 0 to 720896 tid 1 2024-09-16T03:40:13.620 INFO:tasks.rados.rados.0.smithi121.stdout:44: write initial oid 44 2024-09-16T03:40:13.620 INFO:tasks.rados.rados.0.smithi121.stdout:44: seq_num 43 ranges {0=737280} 2024-09-16T03:40:13.625 INFO:tasks.rados.rados.0.smithi121.stdout:44: writing smithi12138290-44 from 0 to 737280 tid 1 2024-09-16T03:40:13.625 INFO:tasks.rados.rados.0.smithi121.stdout:45: write initial oid 45 2024-09-16T03:40:13.625 INFO:tasks.rados.rados.0.smithi121.stdout:45: seq_num 44 ranges {0=753664} 2024-09-16T03:40:13.631 INFO:tasks.rados.rados.0.smithi121.stdout:45: writing smithi12138290-45 from 0 to 753664 tid 1 2024-09-16T03:40:13.631 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.631 INFO:tasks.rados.rados.0.smithi121.stdout:30: finishing write tid 3 to smithi12138290-30 2024-09-16T03:40:13.631 INFO:tasks.rados.rados.0.smithi121.stdout:30: oid 30 updating version 0 to 6 2024-09-16T03:40:13.631 INFO:tasks.rados.rados.0.smithi121.stdout:30: oid 30 version 6 is already newer than 5 2024-09-16T03:40:13.631 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:40:13.631 INFO:tasks.rados.rados.0.smithi121.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:32: finishing write tid 1 to smithi12138290-32 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:32: finishing write tid 2 to smithi12138290-32 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:32: finishing write tid 3 to smithi12138290-32 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:32: oid 32 updating version 0 to 7 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:32: oid 32 updating version 7 to 8 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 32 v 8 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:30: done (15 left) 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:32: done (14 left) 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:46: write initial oid 46 2024-09-16T03:40:13.632 INFO:tasks.rados.rados.0.smithi121.stdout:46: seq_num 45 ranges {0=770048} 2024-09-16T03:40:13.637 INFO:tasks.rados.rados.0.smithi121.stdout:46: writing smithi12138290-46 from 0 to 770048 tid 1 2024-09-16T03:40:13.637 INFO:tasks.rados.rados.0.smithi121.stdout:47: write initial oid 47 2024-09-16T03:40:13.637 INFO:tasks.rados.rados.0.smithi121.stdout:33: finishing write tid 1 to smithi12138290-33 2024-09-16T03:40:13.637 INFO:tasks.rados.rados.0.smithi121.stdout:33: finishing write tid 2 to smithi12138290-33 2024-09-16T03:40:13.637 INFO:tasks.rados.rados.0.smithi121.stdout:47: seq_num 46 ranges {0=786432} 2024-09-16T03:40:13.643 INFO:tasks.rados.rados.0.smithi121.stdout:47: writing smithi12138290-47 from 0 to 786432 tid 1 2024-09-16T03:40:13.643 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.643 INFO:tasks.rados.rados.0.smithi121.stdout:33: finishing write tid 3 to smithi12138290-33 2024-09-16T03:40:13.643 INFO:tasks.rados.rados.0.smithi121.stdout:33: oid 33 updating version 0 to 2 2024-09-16T03:40:13.643 INFO:tasks.rados.rados.0.smithi121.stdout:33: oid 33 version 2 is already newer than 1 2024-09-16T03:40:13.643 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 33 v 2 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:40:13.643 INFO:tasks.rados.rados.0.smithi121.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-16T03:40:13.643 INFO:tasks.rados.rados.0.smithi121.stdout:35: finishing write tid 1 to smithi12138290-35 2024-09-16T03:40:13.643 INFO:tasks.rados.rados.0.smithi121.stdout:35: finishing write tid 2 to smithi12138290-35 2024-09-16T03:40:13.643 INFO:tasks.rados.rados.0.smithi121.stdout:36: finishing write tid 1 to smithi12138290-36 2024-09-16T03:40:13.644 INFO:tasks.rados.rados.0.smithi121.stdout:35: finishing write tid 3 to smithi12138290-35 2024-09-16T03:40:13.644 INFO:tasks.rados.rados.0.smithi121.stdout:35: oid 35 updating version 0 to 10 2024-09-16T03:40:13.644 INFO:tasks.rados.rados.0.smithi121.stdout:35: oid 35 version 10 is already newer than 9 2024-09-16T03:40:13.644 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 35 v 10 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:40:13.644 INFO:tasks.rados.rados.0.smithi121.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:40:13.644 INFO:tasks.rados.rados.0.smithi121.stdout:36: finishing write tid 2 to smithi12138290-36 2024-09-16T03:40:13.644 INFO:tasks.rados.rados.0.smithi121.stdout:33: done (15 left) 2024-09-16T03:40:13.644 INFO:tasks.rados.rados.0.smithi121.stdout:35: done (14 left) 2024-09-16T03:40:13.644 INFO:tasks.rados.rados.0.smithi121.stdout:48: write initial oid 48 2024-09-16T03:40:13.644 INFO:tasks.rados.rados.0.smithi121.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-09-16T03:40:13.645 INFO:tasks.rados.rados.0.smithi121.stdout:48: writing smithi12138290-48 from 0 to 425984 tid 1 2024-09-16T03:40:13.647 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:13.645+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:13.648 INFO:tasks.rados.rados.0.smithi121.stdout:48: writing smithi12138290-48 from 425984 to 802816 tid 2 2024-09-16T03:40:13.648 INFO:tasks.rados.rados.0.smithi121.stdout:49: write initial oid 49 2024-09-16T03:40:13.648 INFO:tasks.rados.rados.0.smithi121.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-09-16T03:40:13.651 INFO:tasks.rados.rados.0.smithi121.stdout:49: writing smithi12138290-49 from 0 to 442368 tid 1 2024-09-16T03:40:13.656 INFO:tasks.rados.rados.0.smithi121.stdout:49: writing smithi12138290-49 from 442368 to 819200 tid 2 2024-09-16T03:40:13.656 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 16 2024-09-16T03:40:13.656 INFO:tasks.rados.rados.0.smithi121.stdout:36: finishing write tid 3 to smithi12138290-36 2024-09-16T03:40:13.656 INFO:tasks.rados.rados.0.smithi121.stdout:36: oid 36 updating version 0 to 3 2024-09-16T03:40:13.656 INFO:tasks.rados.rados.0.smithi121.stdout:36: oid 36 updating version 3 to 4 2024-09-16T03:40:13.656 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 36 v 4 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:40:13.656 INFO:tasks.rados.rados.0.smithi121.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-16T03:40:13.656 INFO:tasks.rados.rados.0.smithi121.stdout:37: finishing write tid 1 to smithi12138290-37 2024-09-16T03:40:13.656 INFO:tasks.rados.rados.0.smithi121.stdout:37: finishing write tid 2 to smithi12138290-37 2024-09-16T03:40:13.656 INFO:tasks.rados.rados.0.smithi121.stdout:37: finishing write tid 3 to smithi12138290-37 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:37: oid 37 updating version 0 to 3 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:37: oid 37 updating version 3 to 4 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 37 v 4 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:38: finishing write tid 1 to smithi12138290-38 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:38: finishing write tid 2 to smithi12138290-38 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:38: finishing write tid 3 to smithi12138290-38 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:38: oid 38 updating version 0 to 7 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:38: oid 38 updating version 7 to 8 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 38 v 8 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:39: finishing write tid 1 to smithi12138290-39 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:39: finishing write tid 2 to smithi12138290-39 2024-09-16T03:40:13.657 INFO:tasks.rados.rados.0.smithi121.stdout:39: finishing write tid 3 to smithi12138290-39 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:39: oid 39 updating version 0 to 5 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:39: oid 39 updating version 5 to 6 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:40: finishing write tid 1 to smithi12138290-40 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:40: finishing write tid 2 to smithi12138290-40 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:40: finishing write tid 3 to smithi12138290-40 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:40: oid 40 updating version 0 to 2 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:40: oid 40 version 2 is already newer than 1 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 40 v 2 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:41: finishing write tid 1 to smithi12138290-41 2024-09-16T03:40:13.658 INFO:tasks.rados.rados.0.smithi121.stdout:41: finishing write tid 2 to smithi12138290-41 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:41: finishing write tid 3 to smithi12138290-41 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:41: oid 41 updating version 0 to 5 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:41: oid 41 updating version 5 to 6 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:42: finishing write tid 1 to smithi12138290-42 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:36: done (15 left) 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:37: done (14 left) 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:38: done (13 left) 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:39: done (12 left) 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:40: done (11 left) 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:41: done (10 left) 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:50: write initial oid 50 2024-09-16T03:40:13.659 INFO:tasks.rados.rados.0.smithi121.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-09-16T03:40:13.660 INFO:tasks.rados.rados.0.smithi121.stdout:50: writing smithi12138290-50 from 0 to 458752 tid 1 2024-09-16T03:40:13.660 INFO:tasks.rados.rados.0.smithi121.stdout:50: writing smithi12138290-50 from 458752 to 835584 tid 2 2024-09-16T03:40:13.661 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 26 current snap is 0 2024-09-16T03:40:13.661 INFO:tasks.rados.rados.0.smithi121.stdout:51: seq_num 50 ranges {425984=475136,901120=376832} 2024-09-16T03:40:13.666 INFO:tasks.rados.rados.0.smithi121.stdout:51: writing smithi12138290-26 from 425984 to 901120 tid 1 2024-09-16T03:40:13.670 INFO:tasks.rados.rados.0.smithi121.stdout:51: writing smithi12138290-26 from 901120 to 1277952 tid 2 2024-09-16T03:40:13.670 INFO:tasks.rados.rados.0.smithi121.stdout:52: delete oid 39 current snap is 0 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:42: finishing write tid 2 to smithi12138290-42 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:42: finishing write tid 3 to smithi12138290-42 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:42: oid 42 updating version 0 to 7 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:42: oid 42 updating version 7 to 8 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 42 v 8 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:43: finishing write tid 1 to smithi12138290-43 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:43: finishing write tid 2 to smithi12138290-43 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:43: finishing write tid 3 to smithi12138290-43 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:43: oid 43 updating version 0 to 9 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:43: oid 43 updating version 9 to 10 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 43 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-16T03:40:13.671 INFO:tasks.rados.rados.0.smithi121.stdout:44: finishing write tid 1 to smithi12138290-44 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:44: finishing write tid 2 to smithi12138290-44 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:45: finishing write tid 1 to smithi12138290-45 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:44: finishing write tid 3 to smithi12138290-44 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:44: oid 44 updating version 0 to 11 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:44: oid 44 updating version 11 to 12 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:45: finishing write tid 2 to smithi12138290-45 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:45: finishing write tid 3 to smithi12138290-45 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:45: oid 45 updating version 0 to 14 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:45: oid 45 version 14 is already newer than 13 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 45 v 14 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:40:13.672 INFO:tasks.rados.rados.0.smithi121.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:46: finishing write tid 1 to smithi12138290-46 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:46: finishing write tid 2 to smithi12138290-46 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:46: finishing write tid 3 to smithi12138290-46 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:46: oid 46 updating version 0 to 1 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:46: oid 46 updating version 1 to 2 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 46 v 2 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:47: finishing write tid 1 to smithi12138290-47 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:47: finishing write tid 2 to smithi12138290-47 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:48: finishing write tid 1 to smithi12138290-48 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:47: finishing write tid 3 to smithi12138290-47 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:47: oid 47 updating version 0 to 5 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:47: oid 47 updating version 5 to 6 2024-09-16T03:40:13.673 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:48: finishing write tid 2 to smithi12138290-48 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:48: finishing write tid 3 to smithi12138290-48 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:49: finishing write tid 1 to smithi12138290-49 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:48: finishing write tid 4 to smithi12138290-48 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:48: oid 48 updating version 0 to 7 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:48: oid 48 version 7 is already newer than 5 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:48: oid 48 version 7 is already newer than 6 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 48 v 7 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:49: finishing write tid 2 to smithi12138290-49 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:49: finishing write tid 3 to smithi12138290-49 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:49: finishing write tid 4 to smithi12138290-49 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:49: oid 49 updating version 0 to 3 2024-09-16T03:40:13.674 INFO:tasks.rados.rados.0.smithi121.stdout:49: oid 49 updating version 3 to 4 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:49: oid 49 updating version 4 to 5 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 49 v 5 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:50: finishing write tid 1 to smithi12138290-50 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:50: finishing write tid 2 to smithi12138290-50 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:50: finishing write tid 3 to smithi12138290-50 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:50: finishing write tid 4 to smithi12138290-50 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:50: oid 50 updating version 0 to 3 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:50: oid 50 updating version 3 to 5 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:50: oid 50 version 5 is already newer than 4 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 50 v 5 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-16T03:40:13.675 INFO:tasks.rados.rados.0.smithi121.stdout:51: finishing write tid 1 to smithi12138290-26 2024-09-16T03:40:13.676 INFO:tasks.rados.rados.0.smithi121.stdout:42: done (12 left) 2024-09-16T03:40:13.676 INFO:tasks.rados.rados.0.smithi121.stdout:43: done (11 left) 2024-09-16T03:40:13.676 INFO:tasks.rados.rados.0.smithi121.stdout:44: done (10 left) 2024-09-16T03:40:13.676 INFO:tasks.rados.rados.0.smithi121.stdout:45: done (9 left) 2024-09-16T03:40:13.676 INFO:tasks.rados.rados.0.smithi121.stdout:46: done (8 left) 2024-09-16T03:40:13.677 INFO:tasks.rados.rados.0.smithi121.stdout:47: done (7 left) 2024-09-16T03:40:13.677 INFO:tasks.rados.rados.0.smithi121.stdout:48: done (6 left) 2024-09-16T03:40:13.677 INFO:tasks.rados.rados.0.smithi121.stdout:49: done (5 left) 2024-09-16T03:40:13.677 INFO:tasks.rados.rados.0.smithi121.stdout:50: done (4 left) 2024-09-16T03:40:13.677 INFO:tasks.rados.rados.0.smithi121.stdout:52: done (3 left) 2024-09-16T03:40:13.677 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 49 current snap is 0 2024-09-16T03:40:13.677 INFO:tasks.rados.rados.0.smithi121.stdout:53: seq_num 52 ranges {819200=507904,1327104=376832} 2024-09-16T03:40:13.680 INFO:tasks.rados.rados.0.smithi121.stdout:53: writing smithi12138290-49 from 819200 to 1327104 tid 1 2024-09-16T03:40:13.684 INFO:tasks.rados.rados.0.smithi121.stdout:53: writing smithi12138290-49 from 1327104 to 1703936 tid 2 2024-09-16T03:40:13.684 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 2 current snap is 0 2024-09-16T03:40:13.684 INFO:tasks.rados.rados.0.smithi121.stdout:51: finishing write tid 2 to smithi12138290-26 2024-09-16T03:40:13.684 INFO:tasks.rados.rados.0.smithi121.stdout:51: finishing write tid 3 to smithi12138290-26 2024-09-16T03:40:13.684 INFO:tasks.rados.rados.0.smithi121.stdout:54: seq_num 53 ranges {32768=524288,557056=376832} 2024-09-16T03:40:13.687 INFO:tasks.rados.rados.0.smithi121.stdout:54: writing smithi12138290-2 from 32768 to 557056 tid 1 2024-09-16T03:40:13.690 INFO:tasks.rados.rados.0.smithi121.stdout:54: writing smithi12138290-2 from 557056 to 933888 tid 2 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:55: rmattr oid 4 current snap is 0 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:51: finishing write tid 4 to smithi12138290-26 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:51: oid 26 updating version 0 to 16 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:51: oid 26 version 16 is already newer than 15 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:51: oid 26 updating version 16 to 17 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 26 v 17 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:51: left oid 26 (ObjNum 50 snap 0 seq_num 50) 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:51: done (5 left) 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:55: done (4 left) 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:56: rollback oid 46 current snap is 0 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:56: done (4 left) 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:57: delete oid 36 current snap is 0 2024-09-16T03:40:13.691 INFO:tasks.rados.rados.0.smithi121.stdout:53: finishing write tid 1 to smithi12138290-49 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:53: finishing write tid 2 to smithi12138290-49 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:53: finishing write tid 3 to smithi12138290-49 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:53: finishing write tid 4 to smithi12138290-49 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:53: oid 49 updating version 0 to 6 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:53: oid 49 updating version 6 to 8 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:53: oid 49 version 8 is already newer than 7 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 49 v 8 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:53: left oid 49 (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:53: done (4 left) 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:57: done (3 left) 2024-09-16T03:40:13.692 INFO:tasks.rados.rados.0.smithi121.stdout:58: setattr oid 12 current snap is 0 2024-09-16T03:40:13.693 INFO:tasks.rados.rados.0.smithi121.stdout:54: finishing write tid 1 to smithi12138290-2 2024-09-16T03:40:13.694 INFO:tasks.rados.rados.0.smithi121.stdout:59: rollback oid 5 current snap is 0 2024-09-16T03:40:13.694 INFO:tasks.rados.rados.0.smithi121.stdout:59: done (4 left) 2024-09-16T03:40:13.694 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 37 current snap is 0 2024-09-16T03:40:13.694 INFO:tasks.rados.rados.0.smithi121.stdout:60: seq_num 55 ranges {606208=557056,1163264=376832} 2024-09-16T03:40:13.700 INFO:tasks.rados.rados.0.smithi121.stdout:60: writing smithi12138290-37 from 606208 to 1163264 tid 1 2024-09-16T03:40:13.702 INFO:tasks.rados.rados.0.smithi121.stdout:60: writing smithi12138290-37 from 1163264 to 1540096 tid 2 2024-09-16T03:40:13.702 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 15 current snap is 0 2024-09-16T03:40:13.703 INFO:tasks.rados.rados.0.smithi121.stdout:54: finishing write tid 2 to smithi12138290-2 2024-09-16T03:40:13.703 INFO:tasks.rados.rados.0.smithi121.stdout:54: finishing write tid 3 to smithi12138290-2 2024-09-16T03:40:13.703 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 7 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:40:13.703 INFO:tasks.rados.rados.0.smithi121.stdout:54: finishing write tid 4 to smithi12138290-2 2024-09-16T03:40:13.703 INFO:tasks.rados.rados.0.smithi121.stdout:54: oid 2 updating version 0 to 19 2024-09-16T03:40:13.703 INFO:tasks.rados.rados.0.smithi121.stdout:54: oid 2 updating version 19 to 20 2024-09-16T03:40:13.703 INFO:tasks.rados.rados.0.smithi121.stdout:54: oid 2 version 20 is already newer than 18 2024-09-16T03:40:13.703 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 2 v 20 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-09-16T03:40:13.703 INFO:tasks.rados.rados.0.smithi121.stdout:54: left oid 2 (ObjNum 53 snap 0 seq_num 53) 2024-09-16T03:40:13.703 INFO:tasks.rados.rados.0.smithi121.stdout:61: seq_num 56 ranges {245760=573440,819200=376832} 2024-09-16T03:40:13.706 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:13.706 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:13.707 INFO:tasks.rados.rados.0.smithi121.stdout:61: writing smithi12138290-15 from 245760 to 819200 tid 1 2024-09-16T03:40:13.709 INFO:tasks.rados.rados.0.smithi121.stdout:61: writing smithi12138290-15 from 819200 to 1196032 tid 2 2024-09-16T03:40:13.709 INFO:tasks.rados.rados.0.smithi121.stdout:54: done (5 left) 2024-09-16T03:40:13.709 INFO:tasks.rados.rados.0.smithi121.stdout:58: done (4 left) 2024-09-16T03:40:13.709 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 6 current snap is 0 2024-09-16T03:40:13.710 INFO:tasks.rados.rados.0.smithi121.stdout:60: finishing write tid 1 to smithi12138290-37 2024-09-16T03:40:13.710 INFO:tasks.rados.rados.0.smithi121.stdout:62: seq_num 57 ranges {98304=589824,688128=376832} 2024-09-16T03:40:13.713 INFO:tasks.rados.rados.0.smithi121.stdout:62: writing smithi12138290-6 from 98304 to 688128 tid 1 2024-09-16T03:40:13.716 INFO:tasks.rados.rados.0.smithi121.stdout:62: writing smithi12138290-6 from 688128 to 1064960 tid 2 2024-09-16T03:40:13.716 INFO:tasks.rados.rados.0.smithi121.stdout:63: copy_from oid 17 from oid 36 current snap is 0 2024-09-16T03:40:13.716 INFO:tasks.rados.rados.0.smithi121.stdout:64: copy_from oid 24 from oid 32 current snap is 0 2024-09-16T03:40:13.716 INFO:tasks.rados.rados.0.smithi121.stdout:60: finishing write tid 2 to smithi12138290-37 2024-09-16T03:40:13.716 INFO:tasks.rados.rados.0.smithi121.stdout:60: finishing write tid 3 to smithi12138290-37 2024-09-16T03:40:13.716 INFO:tasks.rados.rados.0.smithi121.stdout:60: finishing write tid 4 to smithi12138290-37 2024-09-16T03:40:13.716 INFO:tasks.rados.rados.0.smithi121.stdout:60: oid 37 updating version 0 to 10 2024-09-16T03:40:13.717 INFO:tasks.rados.rados.0.smithi121.stdout:60: oid 37 version 10 is already newer than 9 2024-09-16T03:40:13.717 INFO:tasks.rados.rados.0.smithi121.stdout:60: oid 37 version 10 is already newer than 8 2024-09-16T03:40:13.717 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 37 v 10 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-16T03:40:13.717 INFO:tasks.rados.rados.0.smithi121.stdout:60: left oid 37 (ObjNum 55 snap 0 seq_num 55) 2024-09-16T03:40:13.717 INFO:tasks.rados.rados.0.smithi121.stdout:61: finishing write tid 1 to smithi12138290-15 2024-09-16T03:40:13.717 INFO:tasks.rados.rados.0.smithi121.stdout:60: done (6 left) 2024-09-16T03:40:13.717 INFO:tasks.rados.rados.0.smithi121.stdout:65: delete oid 20 current snap is 0 2024-09-16T03:40:13.717 INFO:tasks.rados.rados.0.smithi121.stdout:61: finishing write tid 2 to smithi12138290-15 2024-09-16T03:40:13.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:40:13.719 INFO:tasks.rados.rados.0.smithi121.stdout:61: finishing write tid 3 to smithi12138290-15 2024-09-16T03:40:13.720 INFO:tasks.rados.rados.0.smithi121.stdout:62: finishing write tid 1 to smithi12138290-6 2024-09-16T03:40:13.720 INFO:tasks.rados.rados.0.smithi121.stdout:61: finishing write tid 4 to smithi12138290-15 2024-09-16T03:40:13.720 INFO:tasks.rados.rados.0.smithi121.stdout:61: oid 15 updating version 0 to 8 2024-09-16T03:40:13.720 INFO:tasks.rados.rados.0.smithi121.stdout:61: oid 15 version 8 is already newer than 7 2024-09-16T03:40:13.720 INFO:tasks.rados.rados.0.smithi121.stdout:61: oid 15 updating version 8 to 9 2024-09-16T03:40:13.720 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 15 v 9 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-09-16T03:40:13.720 INFO:tasks.rados.rados.0.smithi121.stdout:61: left oid 15 (ObjNum 56 snap 0 seq_num 56) 2024-09-16T03:40:13.720 INFO:tasks.rados.rados.0.smithi121.stdout:62: finishing write tid 2 to smithi12138290-6 2024-09-16T03:40:13.720 INFO:tasks.rados.rados.0.smithi121.stdout:62: finishing write tid 3 to smithi12138290-6 2024-09-16T03:40:13.721 INFO:tasks.rados.rados.0.smithi121.stdout:61: done (6 left) 2024-09-16T03:40:13.721 INFO:tasks.rados.rados.0.smithi121.stdout:65: done (5 left) 2024-09-16T03:40:13.721 INFO:tasks.rados.rados.0.smithi121.stdout:66: read oid 38 snap -1 2024-09-16T03:40:13.721 INFO:tasks.rados.rados.0.smithi121.stdout:66: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T03:40:13.721 INFO:tasks.rados.rados.0.smithi121.stdout:67: delete oid 5 current snap is 0 2024-09-16T03:40:13.722 INFO:tasks.rados.rados.0.smithi121.stdout:63: finishing copy_from to smithi12138290-17 2024-09-16T03:40:13.722 INFO:tasks.rados.rados.0.smithi121.stdout:63: got expected ENOENT (src dne) 2024-09-16T03:40:13.722 INFO:tasks.rados.rados.0.smithi121.stdout:63: finishing copy_from racing read to smithi12138290-17 2024-09-16T03:40:13.722 INFO:tasks.rados.rados.0.smithi121.stdout:62: finishing write tid 4 to smithi12138290-6 2024-09-16T03:40:13.722 INFO:tasks.rados.rados.0.smithi121.stdout:62: oid 6 updating version 0 to 6 2024-09-16T03:40:13.722 INFO:tasks.rados.rados.0.smithi121.stdout:62: oid 6 updating version 6 to 7 2024-09-16T03:40:13.722 INFO:tasks.rados.rados.0.smithi121.stdout:62: oid 6 updating version 7 to 8 2024-09-16T03:40:13.722 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 6 v 8 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-09-16T03:40:13.722 INFO:tasks.rados.rados.0.smithi121.stdout:62: left oid 6 (ObjNum 57 snap 0 seq_num 57) 2024-09-16T03:40:13.723 INFO:tasks.rados.rados.0.smithi121.stdout:64: finishing copy_from racing read to smithi12138290-24 2024-09-16T03:40:13.724 INFO:tasks.rados.rados.0.smithi121.stdout:62: done (6 left) 2024-09-16T03:40:13.724 INFO:tasks.rados.rados.0.smithi121.stdout:63: done (5 left) 2024-09-16T03:40:13.724 INFO:tasks.rados.rados.0.smithi121.stdout:67: done (4 left) 2024-09-16T03:40:13.724 INFO:tasks.rados.rados.0.smithi121.stdout:68: rollback oid 43 current snap is 0 2024-09-16T03:40:13.724 INFO:tasks.rados.rados.0.smithi121.stdout:68: done (4 left) 2024-09-16T03:40:13.724 INFO:tasks.rados.rados.0.smithi121.stdout:69: delete oid 22 current snap is 0 2024-09-16T03:40:13.725 INFO:tasks.rados.rados.0.smithi121.stdout:64: finishing copy_from to smithi12138290-24 2024-09-16T03:40:13.725 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 24 v 8 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:40:13.726 INFO:tasks.rados.rados.0.smithi121.stdout:64: done (4 left) 2024-09-16T03:40:13.726 INFO:tasks.rados.rados.0.smithi121.stdout:69: done (3 left) 2024-09-16T03:40:13.726 INFO:tasks.rados.rados.0.smithi121.stdout:70: read oid 47 snap -1 2024-09-16T03:40:13.726 INFO:tasks.rados.rados.0.smithi121.stdout:70: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-16T03:40:13.726 INFO:tasks.rados.rados.0.smithi121.stdout:71: rollback oid 40 current snap is 0 2024-09-16T03:40:13.727 INFO:tasks.rados.rados.0.smithi121.stdout:71: done (4 left) 2024-09-16T03:40:13.727 INFO:tasks.rados.rados.0.smithi121.stdout:72: read oid 18 snap -1 2024-09-16T03:40:13.727 INFO:tasks.rados.rados.0.smithi121.stdout:72: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T03:40:13.727 INFO:tasks.rados.rados.0.smithi121.stdout:73: copy_from oid 32 from oid 4 current snap is 0 2024-09-16T03:40:13.727 INFO:tasks.rados.rados.0.smithi121.stdout:74: snap_create 2024-09-16T03:40:13.728 INFO:tasks.rados.rados.0.smithi121.stdout:66: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T03:40:13.745 INFO:tasks.rados.rados.0.smithi121.stdout:72: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T03:40:13.748 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:13.746+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:13.753 INFO:tasks.rados.rados.0.smithi121.stdout:70: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-16T03:40:13.774 INFO:tasks.rados.rados.0.smithi121.stdout:27: finishing write tid 1 to smithi12138290-27 2024-09-16T03:40:13.774 INFO:tasks.rados.rados.0.smithi121.stdout:27: finishing write tid 2 to smithi12138290-27 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:34: finishing write tid 1 to smithi12138290-34 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:34: finishing write tid 2 to smithi12138290-34 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:27: finishing write tid 3 to smithi12138290-27 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:27: oid 27 updating version 0 to 2 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:27: oid 27 version 2 is already newer than 1 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 27 v 2 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:34: finishing write tid 3 to smithi12138290-34 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:34: oid 34 updating version 0 to 4 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:34: oid 34 version 4 is already newer than 3 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 34 v 4 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:40:13.775 INFO:tasks.rados.rados.0.smithi121.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-16T03:40:13.828 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:13.828 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:13.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:40:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:13.842 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 34, 2024-09-16T03:40:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "ecf7a42a-e941-4b24-8693-0ff8b6eb41b1", 2024-09-16T03:40:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T03:39:42.686123+0000", 2024-09-16T03:40:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T03:40:13.497006+0000", 2024-09-16T03:40:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2024-09-16T03:39:56.235015+0000", 2024-09-16T03:40:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2024-09-16T03:39:46.749797+0000", 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5806080, 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "nodeep-scrub", 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "noscrub", 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 7, 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T03:40:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 3, 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 16, 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "jewel", 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "squid", 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_crimson": false, 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2024-09-16T03:40:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".mgr", 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T03:39:56.952065+0000", 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "is_stretch_pool": false, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T03:40:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T03:40:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "26", 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T03:40:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T03:40:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2024-09-16T03:40:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_max": 32, 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": {} 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "read_balance": { 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "score_type": "Fair distribution", 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "score_acting": 15.380000114440918, 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "score_stable": 15.380000114440918, 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "optimal_score": 0.12999999523162842, 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "raw_score_acting": 2, 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "raw_score_stable": 2, 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity_weighted": 1, 2024-09-16T03:40:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "average_primary_affinity": 1, 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "average_primary_affinity_weighted": 1 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "rbd", 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T03:39:57.213209+0000", 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T03:40:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "is_stretch_pool": false, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 8, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 8, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 8, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 8, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 8, 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T03:40:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "27", 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 2, 2024-09-16T03:40:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 27, 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:40:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:40:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "read_balance": { 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "score_type": "Fair distribution", 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "score_acting": 6.0199999809265137, 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "score_stable": 6.0199999809265137, 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "optimal_score": 0.81000000238418579, 2024-09-16T03:40:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "raw_score_acting": 4.880000114440918, 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "raw_score_stable": 4.880000114440918, 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity_weighted": 1, 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "average_primary_affinity": 1, 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "average_primary_affinity_weighted": 1 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T03:40:09.501089+0000", 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 3, 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 7, 2024-09-16T03:40:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 5, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 1, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "is_stretch_pool": false, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2024-09-16T03:40:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "34", 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T03:40:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T03:40:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "shecprofile", 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:40:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 16384, 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "8820e1f9-b619-44a5-abb1-7cfa97fcef00", 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/2834882614", 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/2834882614", 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/2834882614", 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/2834882614", 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T03:40:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "0f58ae45-e2ff-4c4f-9875-7d34ec312cbe", 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 22, 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6808", 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6809", 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6810", 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6811", 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6814", 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6815", 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6812", 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6813", 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.103:6809/2045143946", 2024-09-16T03:40:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.103:6811/2045143946", 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.103:6815/2045143946", 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.103:6813/2045143946", 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b369fe92-86b8-45b3-a418-024f0e4becd7", 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 0, 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6816", 2024-09-16T03:40:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6817", 2024-09-16T03:40:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6818", 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6819", 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6822", 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6823", 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6820", 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6821", 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.119:6817/273257286", 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.119:6819/273257286", 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.119:6823/273257286", 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.119:6821/273257286", 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b9521bf1-46ef-402c-8472-c1b476280b23", 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 0, 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6809", 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6811", 2024-09-16T03:40:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6813", 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6815", 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6821", 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6823", 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6817", 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6819", 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.121:6811/278480145", 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.121:6815/278480145", 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.121:6823/278480145", 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.121:6819/278480145", 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T03:40:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "15bec6a1-fc13-4822-8cfa-899b83886c10", 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 0, 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/3302785479", 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/3302785479", 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/3302785479", 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/3302785479", 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "e9b234ec-bfa4-4f72-adab-295e067b2e61", 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 0, 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6800", 2024-09-16T03:40:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6801", 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6802", 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6803", 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6806", 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6807", 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6804", 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6805", 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.103:6801/3765230361", 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.103:6803/3765230361", 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.103:6807/3765230361", 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.103:6805/3765230361", 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "6d4ae9fb-8d8f-451d-a12f-1265302b03ab", 2024-09-16T03:40:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 22, 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6824", 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6825", 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6826", 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6827", 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6830", 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6831", 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6828", 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6829", 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.119:6825/446165121", 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.119:6827/446165121", 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.119:6831/446165121", 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.119:6829/446165121", 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "2c9456ed-00ec-46aa-814e-f85eb441fc4f", 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 0, 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6808", 2024-09-16T03:40:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6810", 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6812", 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6814", 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6820", 2024-09-16T03:40:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6822", 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6816", 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6818", 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.121:6810/1304791608", 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.121:6814/1304791608", 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.121:6822/1304791608", 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.121:6818/1304791608", 2024-09-16T03:40:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 8, 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b618a02a-8628-48f5-bb5f-f7591aafa118", 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2024-09-16T03:40:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2024-09-16T03:40:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2024-09-16T03:40:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/2630853906", 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/2630853906", 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/2630853906", 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/2630853906", 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 9, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "dc3e5be7-f012-47e6-ba35-0e3354ca0c18", 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6816", 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6817", 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6818", 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6819", 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6822", 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6823", 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6820", 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6821", 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.103:6817/2702548675", 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.103:6819/2702548675", 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.103:6823/2702548675", 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.103:6821/2702548675", 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 10, 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "4e70a7a8-7ba8-42ab-a45a-e0564b561619", 2024-09-16T03:40:13.922 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 22, 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.923 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6808", 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6809", 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.924 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.925 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6810", 2024-09-16T03:40:13.925 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:13.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6811", 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6814", 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.926 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6815", 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6812", 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6813", 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.119:6809/205142394", 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.119:6811/205142394", 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.119:6815/205142394", 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.119:6813/205142394", 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.928 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 11, 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "d16bc493-335f-41c2-bd68-cbad93c49a5d", 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.929 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6824", 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6825", 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:13.930 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6826", 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6827", 2024-09-16T03:40:13.931 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6830", 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6831", 2024-09-16T03:40:13.932 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6828", 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6829", 2024-09-16T03:40:13.933 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.121:6825/1551091359", 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.121:6827/1551091359", 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.121:6831/1551091359", 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.121:6829/1551091359", 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 12, 2024-09-16T03:40:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1", 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.935 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.938 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/890992293", 2024-09-16T03:40:13.939 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/890992293", 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/890992293", 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/890992293", 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 13, 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "61e9204b-8269-4c35-b409-5cfbd5e6abbb", 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.940 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6824", 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:13.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6825", 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6826", 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6827", 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.943 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6830", 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6831", 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.944 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6828", 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6829", 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.103:6825/3905392668", 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.103:6827/3905392668", 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.103:6831/3905392668", 2024-09-16T03:40:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.103:6829/3905392668", 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 14, 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "0bd5338e-eb99-44cc-9f5f-1c8ef903d02c", 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0.5100555419921875, 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 21, 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6800", 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6801", 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6802", 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6803", 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6806", 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6807", 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6804", 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6805", 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.119:6801/1147733023", 2024-09-16T03:40:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.119:6803/1147733023", 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.119:6807/1147733023", 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.119:6805/1147733023", 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 15, 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f240405e-d268-41cc-bd99-5a2ca71a85a9", 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6800", 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6801", 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6802", 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6803", 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6806", 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6807", 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6804", 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6805", 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.121:6801/3149764080", 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.121:6803/3149764080", 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3149764080", 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.121:6805/3149764080", 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.957 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:53.946705+0000", 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T03:40:13.958 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:53.541012+0000", 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.279946+0000", 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.202293+0000", 2024-09-16T03:40:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:53.783644+0000", 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T03:40:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.098081+0000", 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.053567+0000", 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.262433+0000", 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 8, 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.271589+0000", 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 9, 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.040223+0000", 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 10, 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.048985+0000", 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 11, 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.049910+0000", 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 12, 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.019981+0000", 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 13, 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:53.956380+0000", 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 14, 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:53.072636+0000", 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 15, 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.047013+0000", 2024-09-16T03:40:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [], 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [], 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_primaries": [], 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": {}, 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "range_blocklist": {}, 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2024-09-16T03:40:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2024-09-16T03:40:13.971 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2024-09-16T03:40:13.971 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2024-09-16T03:40:13.971 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2024-09-16T03:40:13.971 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2024-09-16T03:40:13.971 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2024-09-16T03:40:13.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.971 INFO:teuthology.orchestra.run.smithi076.stdout: "shecprofile": { 2024-09-16T03:40:13.971 INFO:teuthology.orchestra.run.smithi076.stdout: "c": "2", 2024-09-16T03:40:13.971 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-device-class": "", 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-num-failure-domains": "0", 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-osds-per-failure-domain": "0", 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-root": "default", 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "4", 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "3", 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "shecprofile", 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "shec", 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "multiple" 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [], 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [], 2024-09-16T03:40:13.972 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2024-09-16T03:40:13.973 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2024-09-16T03:40:13.973 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2024-09-16T03:40:13.973 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2024-09-16T03:40:13.973 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2024-09-16T03:40:13.973 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2024-09-16T03:40:13.973 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2024-09-16T03:40:13.973 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2024-09-16T03:40:13.973 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2024-09-16T03:40:13.973 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:13.973 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:13.975 INFO:tasks.rados.rados.0.smithi121.stdout:73: finishing copy_from racing read to smithi12138290-32 2024-09-16T03:40:13.975 INFO:tasks.rados.rados.0.smithi121.stdout:73: finishing copy_from to smithi12138290-32 2024-09-16T03:40:13.975 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 32 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:40:13.976 INFO:tasks.thrashosds.thrasher:j is {'epoch': 34, 'fsid': 'ecf7a42a-e941-4b24-8693-0ff8b6eb41b1', 'created': '2024-09-16T03:39:42.686123+0000', 'modified': '2024-09-16T03:40:13.497006+0000', 'last_up_change': '2024-09-16T03:39:56.235015+0000', 'last_in_change': '2024-09-16T03:39:46.749797+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T03:39:56.952065+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T03:39:57.213209+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 27, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.019999980926514, 'score_stable': 6.019999980926514, 'optimal_score': 0.8100000023841858, 'raw_score_acting': 4.880000114440918, 'raw_score_stable': 4.880000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T03:40:09.501089+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '8820e1f9-b619-44a5-abb1-7cfa97fcef00', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 2834882614}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 2834882614}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 2834882614}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 2834882614}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 2834882614}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 2834882614}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 2834882614}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 2834882614}]}, 'public_addr': '172.21.15.76:6811/2834882614', 'cluster_addr': '172.21.15.76:6813/2834882614', 'heartbeat_back_addr': '172.21.15.76:6817/2834882614', 'heartbeat_front_addr': '172.21.15.76:6815/2834882614', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0f58ae45-e2ff-4c4f-9875-7d34ec312cbe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6808', 'nonce': 2045143946}, {'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 2045143946}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6810', 'nonce': 2045143946}, {'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 2045143946}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6814', 'nonce': 2045143946}, {'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 2045143946}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6812', 'nonce': 2045143946}, {'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 2045143946}]}, 'public_addr': '172.21.15.103:6809/2045143946', 'cluster_addr': '172.21.15.103:6811/2045143946', 'heartbeat_back_addr': '172.21.15.103:6815/2045143946', 'heartbeat_front_addr': '172.21.15.103:6813/2045143946', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b369fe92-86b8-45b3-a418-024f0e4becd7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6816', 'nonce': 273257286}, {'type': 'v1', 'addr': '172.21.15.119:6817', 'nonce': 273257286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6818', 'nonce': 273257286}, {'type': 'v1', 'addr': '172.21.15.119:6819', 'nonce': 273257286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6822', 'nonce': 273257286}, {'type': 'v1', 'addr': '172.21.15.119:6823', 'nonce': 273257286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6820', 'nonce': 273257286}, {'type': 'v1', 'addr': '172.21.15.119:6821', 'nonce': 273257286}]}, 'public_addr': '172.21.15.119:6817/273257286', 'cluster_addr': '172.21.15.119:6819/273257286', 'heartbeat_back_addr': '172.21.15.119:6823/273257286', 'heartbeat_front_addr': '172.21.15.119:6821/273257286', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'b9521bf1-46ef-402c-8472-c1b476280b23', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6809', 'nonce': 278480145}, {'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 278480145}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6813', 'nonce': 278480145}, {'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 278480145}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6821', 'nonce': 278480145}, {'type': 'v1', 'addr': '172.21.15.121:6823', 'nonce': 278480145}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6817', 'nonce': 278480145}, {'type': 'v1', 'addr': '172.21.15.121:6819', 'nonce': 278480145}]}, 'public_addr': '172.21.15.121:6811/278480145', 'cluster_addr': '172.21.15.121:6815/278480145', 'heartbeat_back_addr': '172.21.15.121:6823/278480145', 'heartbeat_front_addr': '172.21.15.121:6819/278480145', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '15bec6a1-fc13-4822-8cfa-899b83886c10', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 3302785479}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 3302785479}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 3302785479}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 3302785479}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 3302785479}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 3302785479}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 3302785479}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 3302785479}]}, 'public_addr': '172.21.15.76:6803/3302785479', 'cluster_addr': '172.21.15.76:6805/3302785479', 'heartbeat_back_addr': '172.21.15.76:6809/3302785479', 'heartbeat_front_addr': '172.21.15.76:6807/3302785479', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e9b234ec-bfa4-4f72-adab-295e067b2e61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6800', 'nonce': 3765230361}, {'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 3765230361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6802', 'nonce': 3765230361}, {'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 3765230361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6806', 'nonce': 3765230361}, {'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 3765230361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6804', 'nonce': 3765230361}, {'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 3765230361}]}, 'public_addr': '172.21.15.103:6801/3765230361', 'cluster_addr': '172.21.15.103:6803/3765230361', 'heartbeat_back_addr': '172.21.15.103:6807/3765230361', 'heartbeat_front_addr': '172.21.15.103:6805/3765230361', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6d4ae9fb-8d8f-451d-a12f-1265302b03ab', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6824', 'nonce': 446165121}, {'type': 'v1', 'addr': '172.21.15.119:6825', 'nonce': 446165121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6826', 'nonce': 446165121}, {'type': 'v1', 'addr': '172.21.15.119:6827', 'nonce': 446165121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6830', 'nonce': 446165121}, {'type': 'v1', 'addr': '172.21.15.119:6831', 'nonce': 446165121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6828', 'nonce': 446165121}, {'type': 'v1', 'addr': '172.21.15.119:6829', 'nonce': 446165121}]}, 'public_addr': '172.21.15.119:6825/446165121', 'cluster_addr': '172.21.15.119:6827/446165121', 'heartbeat_back_addr': '172.21.15.119:6831/446165121', 'heartbeat_front_addr': '172.21.15.119:6829/446165121', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '2c9456ed-00ec-46aa-814e-f85eb441fc4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6808', 'nonce': 1304791608}, {'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1304791608}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6812', 'nonce': 1304791608}, {'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 1304791608}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6820', 'nonce': 1304791608}, {'type': 'v1', 'addr': '172.21.15.121:6822', 'nonce': 1304791608}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6816', 'nonce': 1304791608}, {'type': 'v1', 'addr': '172.21.15.121:6818', 'nonce': 1304791608}]}, 'public_addr': '172.21.15.121:6810/1304791608', 'cluster_addr': '172.21.15.121:6814/1304791608', 'heartbeat_back_addr': '172.21.15.121:6822/1304791608', 'heartbeat_front_addr': '172.21.15.121:6818/1304791608', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b618a02a-8628-48f5-bb5f-f7591aafa118', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 2630853906}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 2630853906}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 2630853906}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 2630853906}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 2630853906}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 2630853906}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 2630853906}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 2630853906}]}, 'public_addr': '172.21.15.76:6819/2630853906', 'cluster_addr': '172.21.15.76:6821/2630853906', 'heartbeat_back_addr': '172.21.15.76:6825/2630853906', 'heartbeat_front_addr': '172.21.15.76:6823/2630853906', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'dc3e5be7-f012-47e6-ba35-0e3354ca0c18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6816', 'nonce': 2702548675}, {'type': 'v1', 'addr': '172.21.15.103:6817', 'nonce': 2702548675}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6818', 'nonce': 2702548675}, {'type': 'v1', 'addr': '172.21.15.103:6819', 'nonce': 2702548675}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6822', 'nonce': 2702548675}, {'type': 'v1', 'addr': '172.21.15.103:6823', 'nonce': 2702548675}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6820', 'nonce': 2702548675}, {'type': 'v1', 'addr': '172.21.15.103:6821', 'nonce': 2702548675}]}, 'public_addr': '172.21.15.103:6817/2702548675', 'cluster_addr': '172.21.15.103:6819/2702548675', 'heartbeat_back_addr': '172.21.15.103:6823/2702548675', 'heartbeat_front_addr': '172.21.15.103:6821/2702548675', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '4e70a7a8-7ba8-42ab-a45a-e0564b561619', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6808', 'nonce': 205142394}, {'type': 'v1', 'addr': '172.21.15.119:6809', 'nonce': 205142394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6810', 'nonce': 205142394}, {'type': 'v1', 'addr': '172.21.15.119:6811', 'nonce': 205142394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6814', 'nonce': 205142394}, {'type': 'v1', 'addr': '172.21.15.119:6815', 'nonce': 205142394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6812', 'nonce': 205142394}, {'type': 'v1', 'addr': '172.21.15.119:6813', 'nonce': 205142394}]}, 'public_addr': '172.21.15.119:6809/205142394', 'cluster_addr': '172.21.15.119:6811/205142394', 'heartbeat_back_addr': '172.21.15.119:6815/205142394', 'heartbeat_front_addr': '172.21.15.119:6813/205142394', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'd16bc493-335f-41c2-bd68-cbad93c49a5d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6824', 'nonce': 1551091359}, {'type': 'v1', 'addr': '172.21.15.121:6825', 'nonce': 1551091359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6826', 'nonce': 1551091359}, {'type': 'v1', 'addr': '172.21.15.121:6827', 'nonce': 1551091359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6830', 'nonce': 1551091359}, {'type': 'v1', 'addr': '172.21.15.121:6831', 'nonce': 1551091359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6828', 'nonce': 1551091359}, {'type': 'v1', 'addr': '172.21.15.121:6829', 'nonce': 1551091359}]}, 'public_addr': '172.21.15.121:6825/1551091359', 'cluster_addr': '172.21.15.121:6827/1551091359', 'heartbeat_back_addr': '172.21.15.121:6831/1551091359', 'heartbeat_front_addr': '172.21.15.121:6829/1551091359', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 890992293}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 890992293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 890992293}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 890992293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 890992293}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 890992293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 890992293}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 890992293}]}, 'public_addr': '172.21.15.76:6827/890992293', 'cluster_addr': '172.21.15.76:6829/890992293', 'heartbeat_back_addr': '172.21.15.76:6833/890992293', 'heartbeat_front_addr': '172.21.15.76:6831/890992293', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '61e9204b-8269-4c35-b409-5cfbd5e6abbb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6824', 'nonce': 3905392668}, {'type': 'v1', 'addr': '172.21.15.103:6825', 'nonce': 3905392668}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6826', 'nonce': 3905392668}, {'type': 'v1', 'addr': '172.21.15.103:6827', 'nonce': 3905392668}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6830', 'nonce': 3905392668}, {'type': 'v1', 'addr': '172.21.15.103:6831', 'nonce': 3905392668}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6828', 'nonce': 3905392668}, {'type': 'v1', 'addr': '172.21.15.103:6829', 'nonce': 3905392668}]}, 'public_addr': '172.21.15.103:6825/3905392668', 'cluster_addr': '172.21.15.103:6827/3905392668', 'heartbeat_back_addr': '172.21.15.103:6831/3905392668', 'heartbeat_front_addr': '172.21.15.103:6829/3905392668', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '0bd5338e-eb99-44cc-9f5f-1c8ef903d02c', 'up': 1, 'in': 1, 'weight': 0.5100555419921875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6800', 'nonce': 1147733023}, {'type': 'v1', 'addr': '172.21.15.119:6801', 'nonce': 1147733023}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6802', 'nonce': 1147733023}, {'type': 'v1', 'addr': '172.21.15.119:6803', 'nonce': 1147733023}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6806', 'nonce': 1147733023}, {'type': 'v1', 'addr': '172.21.15.119:6807', 'nonce': 1147733023}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6804', 'nonce': 1147733023}, {'type': 'v1', 'addr': '172.21.15.119:6805', 'nonce': 1147733023}]}, 'public_addr': '172.21.15.119:6801/1147733023', 'cluster_addr': '172.21.15.119:6803/1147733023', 'heartbeat_back_addr': '172.21.15.119:6807/1147733023', 'heartbeat_front_addr': '172.21.15.119:6805/1147733023', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'f240405e-d268-41cc-bd99-5a2ca71a85a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6800', 'nonce': 3149764080}, {'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3149764080}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6802', 'nonce': 3149764080}, {'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3149764080}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6806', 'nonce': 3149764080}, {'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3149764080}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6804', 'nonce': 3149764080}, {'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3149764080}]}, 'public_addr': '172.21.15.121:6801/3149764080', 'cluster_addr': '172.21.15.121:6803/3149764080', 'heartbeat_back_addr': '172.21.15.121:6807/3149764080', 'heartbeat_front_addr': '172.21.15.121:6805/3149764080', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:53.946705+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:53.541012+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.279946+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.202293+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:53.783644+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.098081+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.053567+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.262433+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.271589+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.040223+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.048985+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.049910+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.019981+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:53.956380+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:53.072636+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.047013+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'shecprofile': {'c': '2', 'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'k': '4', 'm': '3', 'name': 'shecprofile', 'plugin': 'shec', 'technique': 'multiple'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T03:40:13.977 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:13.977 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:13.972+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:14.075 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:14.073+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:14.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:40:14.176 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:14.174+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:14.187 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:14.187 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:14.196 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:40:14.276 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:14.275+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:14.301 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:40:14.303 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:14.303 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:14.312 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:40:14.314 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-16T03:40:13.502557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":123,"acting":123,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":438932,"kb_used_data":6392,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509510508,"statfs":{"total":1546188226560,"available":1545738760192,"internally_reserved":0,"allocated":6545408,"data_stored":3452744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25518,"internal_metadata":442473554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019298"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.519977+0000","last_change":"2024-09-16T03:40:11.518731+0000","last_active":"2024-09-16T03:40:11.519977+0000","last_peered":"2024-09-16T03:40:11.519977+0000","last_clean":"2024-09-16T03:40:11.519977+0000","last_became_active":"2024-09-16T03:40:11.518147+0000","last_became_peered":"2024-09-16T03:40:11.518147+0000","last_unstale":"2024-09-16T03:40:11.519977+0000","last_undegraded":"2024-09-16T03:40:11.519977+0000","last_fullsized":"2024-09-16T03:40:11.519977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:23.768202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,11,7,1,2],"acting":[0,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.518910+0000","last_change":"2024-09-16T03:40:11.517557+0000","last_active":"2024-09-16T03:40:11.518910+0000","last_peered":"2024-09-16T03:40:11.518910+0000","last_clean":"2024-09-16T03:40:11.518910+0000","last_became_active":"2024-09-16T03:40:11.516892+0000","last_became_peered":"2024-09-16T03:40:11.516892+0000","last_unstale":"2024-09-16T03:40:11.518910+0000","last_undegraded":"2024-09-16T03:40:11.518910+0000","last_fullsized":"2024-09-16T03:40:11.518910+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:10.869436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.519209+0000","last_change":"2024-09-16T03:40:11.516960+0000","last_active":"2024-09-16T03:40:11.519209+0000","last_peered":"2024-09-16T03:40:11.519209+0000","last_clean":"2024-09-16T03:40:11.519209+0000","last_became_active":"2024-09-16T03:40:11.516335+0000","last_became_peered":"2024-09-16T03:40:11.516335+0000","last_unstale":"2024-09-16T03:40:11.519209+0000","last_undegraded":"2024-09-16T03:40:11.519209+0000","last_fullsized":"2024-09-16T03:40:11.519209+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:36.974683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,0,8,2,6],"acting":[12,4,9,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.519560+0000","last_change":"2024-09-16T03:40:11.516561+0000","last_active":"2024-09-16T03:40:11.519560+0000","last_peered":"2024-09-16T03:40:11.519560+0000","last_clean":"2024-09-16T03:40:11.519560+0000","last_became_active":"2024-09-16T03:40:11.515959+0000","last_became_peered":"2024-09-16T03:40:11.515959+0000","last_unstale":"2024-09-16T03:40:11.519560+0000","last_undegraded":"2024-09-16T03:40:11.519560+0000","last_fullsized":"2024-09-16T03:40:11.519560+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:16.786064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,1,9,2,6],"acting":[0,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-16T03:40:10.496857+0000","last_change":"2024-09-16T03:40:10.496857+0000","last_active":"2024-09-16T03:40:10.480737+0000","last_peered":"2024-09-16T03:40:10.480737+0000","last_clean":"2024-09-16T03:40:10.480737+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:40:10.496857+0000","last_undegraded":"2024-09-16T03:40:10.496857+0000","last_fullsized":"2024-09-16T03:40:10.496857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.527376+0000","last_change":"2024-09-16T03:40:11.524759+0000","last_active":"2024-09-16T03:40:11.527376+0000","last_peered":"2024-09-16T03:40:11.527376+0000","last_clean":"2024-09-16T03:40:11.527376+0000","last_became_active":"2024-09-16T03:40:11.523925+0000","last_became_peered":"2024-09-16T03:40:11.523925+0000","last_unstale":"2024-09-16T03:40:11.527376+0000","last_undegraded":"2024-09-16T03:40:11.527376+0000","last_fullsized":"2024-09-16T03:40:11.527376+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:12.813966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.506282+0000","last_change":"2024-09-16T03:40:11.503080+0000","last_active":"2024-09-16T03:40:11.506282+0000","last_peered":"2024-09-16T03:40:11.506282+0000","last_clean":"2024-09-16T03:40:11.506282+0000","last_became_active":"2024-09-16T03:40:11.502278+0000","last_became_peered":"2024-09-16T03:40:11.502278+0000","last_unstale":"2024-09-16T03:40:11.506282+0000","last_undegraded":"2024-09-16T03:40:11.506282+0000","last_fullsized":"2024-09-16T03:40:11.506282+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:37.178777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.517681+0000","last_change":"2024-09-16T03:40:11.516798+0000","last_active":"2024-09-16T03:40:11.517681+0000","last_peered":"2024-09-16T03:40:11.517681+0000","last_clean":"2024-09-16T03:40:11.517681+0000","last_became_active":"2024-09-16T03:40:11.516188+0000","last_became_peered":"2024-09-16T03:40:11.516188+0000","last_unstale":"2024-09-16T03:40:11.517681+0000","last_undegraded":"2024-09-16T03:40:11.517681+0000","last_fullsized":"2024-09-16T03:40:11.517681+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:21.492146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,0,5],"acting":[11,10,1,15,14,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":220,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.495639+0000","last_change":"2024-09-16T03:39:58.261761+0000","last_active":"2024-09-16T03:40:11.495639+0000","last_peered":"2024-09-16T03:40:11.495639+0000","last_clean":"2024-09-16T03:40:11.495639+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:40:11.495639+0000","last_undegraded":"2024-09-16T03:40:11.495639+0000","last_fullsized":"2024-09-16T03:40:11.495639+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.270453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":30,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.494744+0000","last_change":"2024-09-16T03:40:00.965641+0000","last_active":"2024-09-16T03:40:11.494744+0000","last_peered":"2024-09-16T03:40:11.494744+0000","last_clean":"2024-09-16T03:40:11.494744+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:40:11.494744+0000","last_undegraded":"2024-09-16T03:40:11.494744+0000","last_fullsized":"2024-09-16T03:40:11.494744+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:22.133279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.525807+0000","last_change":"2024-09-16T03:40:11.524930+0000","last_active":"2024-09-16T03:40:11.525807+0000","last_peered":"2024-09-16T03:40:11.525807+0000","last_clean":"2024-09-16T03:40:11.525807+0000","last_became_active":"2024-09-16T03:40:11.524110+0000","last_became_peered":"2024-09-16T03:40:11.524110+0000","last_unstale":"2024-09-16T03:40:11.525807+0000","last_undegraded":"2024-09-16T03:40:11.525807+0000","last_fullsized":"2024-09-16T03:40:11.525807+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:29.904001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0,4,8,3,7],"acting":[11,5,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.495052+0000","last_change":"2024-09-16T03:40:00.334268+0000","last_active":"2024-09-16T03:40:11.495052+0000","last_peered":"2024-09-16T03:40:11.495052+0000","last_clean":"2024-09-16T03:40:11.495052+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:40:11.495052+0000","last_undegraded":"2024-09-16T03:40:11.495052+0000","last_fullsized":"2024-09-16T03:40:11.495052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:16.834552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-16T03:40:10.494916+0000","last_change":"2024-09-16T03:40:10.494916+0000","last_active":"2024-09-16T03:40:10.480737+0000","last_peered":"2024-09-16T03:40:10.480737+0000","last_clean":"2024-09-16T03:40:10.480737+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:40:10.494916+0000","last_undegraded":"2024-09-16T03:40:10.494916+0000","last_fullsized":"2024-09-16T03:40:10.494916+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.491667+0000","last_change":"2024-09-16T03:40:00.965529+0000","last_active":"2024-09-16T03:40:11.491667+0000","last_peered":"2024-09-16T03:40:11.491667+0000","last_clean":"2024-09-16T03:40:11.491667+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:40:11.491667+0000","last_undegraded":"2024-09-16T03:40:11.491667+0000","last_fullsized":"2024-09-16T03:40:11.491667+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.759480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-16T03:40:10.495423+0000","last_change":"2024-09-16T03:40:10.495423+0000","last_active":"2024-09-16T03:40:10.480737+0000","last_peered":"2024-09-16T03:40:10.480737+0000","last_clean":"2024-09-16T03:40:10.480737+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:40:10.495423+0000","last_undegraded":"2024-09-16T03:40:10.495423+0000","last_fullsized":"2024-09-16T03:40:10.495423+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"27'2","reported_seq":31,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.495911+0000","last_change":"2024-09-16T03:40:00.334487+0000","last_active":"2024-09-16T03:40:11.495911+0000","last_peered":"2024-09-16T03:40:11.495911+0000","last_clean":"2024-09-16T03:40:11.495911+0000","last_became_active":"2024-09-16T03:39:58.564548+0000","last_became_peered":"2024-09-16T03:39:58.564548+0000","last_unstale":"2024-09-16T03:40:11.495911+0000","last_undegraded":"2024-09-16T03:40:11.495911+0000","last_fullsized":"2024-09-16T03:40:11.495911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:24.220303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.518130+0000","last_change":"2024-09-16T03:40:11.516503+0000","last_active":"2024-09-16T03:40:11.518130+0000","last_peered":"2024-09-16T03:40:11.518130+0000","last_clean":"2024-09-16T03:40:11.518130+0000","last_became_active":"2024-09-16T03:40:11.515529+0000","last_became_peered":"2024-09-16T03:40:11.515529+0000","last_unstale":"2024-09-16T03:40:11.518130+0000","last_undegraded":"2024-09-16T03:40:11.518130+0000","last_fullsized":"2024-09-16T03:40:11.518130+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:16.534515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,0,1,4,12],"acting":[15,8,2,0,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.520938+0000","last_change":"2024-09-16T03:40:11.519688+0000","last_active":"2024-09-16T03:40:11.520938+0000","last_peered":"2024-09-16T03:40:11.520938+0000","last_clean":"2024-09-16T03:40:11.520938+0000","last_became_active":"2024-09-16T03:40:11.518878+0000","last_became_peered":"2024-09-16T03:40:11.518878+0000","last_unstale":"2024-09-16T03:40:11.520938+0000","last_undegraded":"2024-09-16T03:40:11.520938+0000","last_fullsized":"2024-09-16T03:40:11.520938+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:19.506189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7,11,10,2,5],"acting":[8,6,7,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T03:40:10.495935+0000","last_change":"2024-09-16T03:40:01.115542+0000","last_active":"2024-09-16T03:40:10.495935+0000","last_peered":"2024-09-16T03:40:10.495935+0000","last_clean":"2024-09-16T03:40:10.495935+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:40:10.495935+0000","last_undegraded":"2024-09-16T03:40:10.495935+0000","last_fullsized":"2024-09-16T03:40:10.495935+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.045739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-16T03:40:10.496816+0000","last_change":"2024-09-16T03:40:10.496816+0000","last_active":"2024-09-16T03:40:10.480737+0000","last_peered":"2024-09-16T03:40:10.480737+0000","last_clean":"2024-09-16T03:40:10.480737+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:40:10.496816+0000","last_undegraded":"2024-09-16T03:40:10.496816+0000","last_fullsized":"2024-09-16T03:40:10.496816+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.491322+0000","last_change":"2024-09-16T03:40:00.333609+0000","last_active":"2024-09-16T03:40:11.491322+0000","last_peered":"2024-09-16T03:40:11.491322+0000","last_clean":"2024-09-16T03:40:11.491322+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:40:11.491322+0000","last_undegraded":"2024-09-16T03:40:11.491322+0000","last_fullsized":"2024-09-16T03:40:11.491322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.665433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.522259+0000","last_change":"2024-09-16T03:40:11.518801+0000","last_active":"2024-09-16T03:40:11.522259+0000","last_peered":"2024-09-16T03:40:11.522259+0000","last_clean":"2024-09-16T03:40:11.522259+0000","last_became_active":"2024-09-16T03:40:11.518056+0000","last_became_peered":"2024-09-16T03:40:11.518056+0000","last_unstale":"2024-09-16T03:40:11.522259+0000","last_undegraded":"2024-09-16T03:40:11.522259+0000","last_fullsized":"2024-09-16T03:40:11.522259+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:18.196247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,13,7,5,11,6,0],"acting":[9,13,7,5,11,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.495614+0000","last_change":"2024-09-16T03:40:10.261076+0000","last_active":"2024-09-16T03:40:11.495614+0000","last_peered":"2024-09-16T03:40:11.495614+0000","last_clean":"2024-09-16T03:40:11.495614+0000","last_became_active":"2024-09-16T03:40:10.260825+0000","last_became_peered":"2024-09-16T03:40:10.260825+0000","last_unstale":"2024-09-16T03:40:11.495614+0000","last_undegraded":"2024-09-16T03:40:11.495614+0000","last_fullsized":"2024-09-16T03:40:11.495614+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:02.522441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3],"acting":[13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:40:10.480737+0000","last_change":"2024-09-16T03:40:10.480737+0000","last_active":"2024-09-16T03:40:10.480737+0000","last_peered":"2024-09-16T03:40:10.480737+0000","last_clean":"2024-09-16T03:40:10.480737+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:40:10.480737+0000","last_undegraded":"2024-09-16T03:40:10.480737+0000","last_fullsized":"2024-09-16T03:40:10.480737+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T03:40:11.495187+0000","last_change":"2024-09-16T03:40:00.334076+0000","last_active":"2024-09-16T03:40:11.495187+0000","last_peered":"2024-09-16T03:40:11.495187+0000","last_clean":"2024-09-16T03:40:11.495187+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:40:11.495187+0000","last_undegraded":"2024-09-16T03:40:11.495187+0000","last_fullsized":"2024-09-16T03:40:11.495187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.276919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":105,"acting":105,"num_store_stats":14},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":148431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":148431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":23,"seq":98784247815,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":356352,"data_stored":158732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313222,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":307200,"data_stored":127112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,6,7,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":138199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":339968,"data_stored":148431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":344064,"data_stored":148450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[0,1,3,6,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":339968,"data_stored":148431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":356352,"data_stored":158732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":339968,"data_stored":148431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":356352,"data_stored":158732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":888832,"data_stored":687660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":339968,"data_stored":148431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247815,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":344064,"data_stored":148450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,7,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247814,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":148431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,4,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":888832,"data_stored":687660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,5,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:14.315 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [7, 10, 3, 0] 2024-09-16T03:40:14.315 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.0', '7', '10', '3', '0'] 2024-09-16T03:40:14.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.0 7 10 3 0 2024-09-16T03:40:14.377 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:14.376+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:14.417 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:14.417 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:14.426 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:40:14.478 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:14.476+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:14.510 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T03:40:14.511 INFO:tasks.rados.rados.0.smithi121.stdout:27: done (6 left) 2024-09-16T03:40:14.511 INFO:tasks.rados.rados.0.smithi121.stdout:34: done (5 left) 2024-09-16T03:40:14.511 INFO:tasks.rados.rados.0.smithi121.stdout:66: done (4 left) 2024-09-16T03:40:14.511 INFO:tasks.rados.rados.0.smithi121.stdout:70: done (3 left) 2024-09-16T03:40:14.512 INFO:tasks.rados.rados.0.smithi121.stdout:72: done (2 left) 2024-09-16T03:40:14.512 INFO:tasks.rados.rados.0.smithi121.stdout:73: done (1 left) 2024-09-16T03:40:14.512 INFO:tasks.rados.rados.0.smithi121.stdout:74: done (0 left) 2024-09-16T03:40:14.512 INFO:tasks.rados.rados.0.smithi121.stdout:75: snap_remove snap 0 2024-09-16T03:40:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:14.566 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:40:14.578 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:14.577+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:14.679 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:14.678+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:14.711 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:14.711 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:14.720 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:40:14.779 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:14.778+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:14.824 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:14.824 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:14.833 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:40:14.880 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:14.879+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:14.937 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:14.937 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:14.946 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:40:14.981 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:14.979+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:15.049 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:15.050 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:15.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:40:15.102 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:15.100+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:15.162 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:15.162 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:15.170 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:40:15.182 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:15.180+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:15.282 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:15.281+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:15.315 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:15.315 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:15.324 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:40:15.383 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:15.382+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:15.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:15.428 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:15.428 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:15.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:40:15.513 INFO:teuthology.orchestra.run.smithi076.stderr:set 2.0 pg_upmap_items mapping to [7->10,3->0] 2024-09-16T03:40:15.513 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:15.510+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:15.516 INFO:tasks.rados.rados.0.smithi121.stdout:75: done (0 left) 2024-09-16T03:40:15.516 INFO:tasks.rados.rados.0.smithi121.stdout:76: snap_create 2024-09-16T03:40:15.523 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:15.523 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:15.541 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:15.541 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:15.558 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:15.559 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:15.576 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:15.576 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:15.584 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:15.583+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:15.587 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:15.587 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:15.592 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:15.592 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:15.596 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:40:15.610 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:15.610 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:15.634 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:15.634 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:15.651 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:15.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:15.667 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:15.667 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:15.685 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:15.683+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:15.698 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:15.698 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:15.729 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:15.729 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:15.742 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:15.754 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:40:15.758 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:15.758 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:15.773 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:15.774 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:15.786 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:15.784+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:15.791 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:15.791 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:15.808 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:15.809 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:15.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:15.806+0000 7f4ed1c81640 0 -- 172.21.15.76:0/3353139374 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f4ecc220ce0 msgr2=0x7f4ecc2230d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:15.870 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:15.871 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:15.871 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:15.871 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:15.879 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:40:15.887 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:15.884+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:15.984 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:15.984 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:15.984 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:15.984 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:15.984 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:15.988 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:15.986+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:15.992 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:40:16.025 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:16.026 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:16.090 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:16.088+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:16.095 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:16.095 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:16.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:16.095 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-09-16T03:40:16.095 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:16.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:40:16.190 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:16.187+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:16.207 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:16.207 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:16.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:40:16.291 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:16.289+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:16.321 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:16.321 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:16.321 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:16.321 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:16.321 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:16.329 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:40:16.392 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:16.390+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:16.474 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:16.474 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:16.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:16.474 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:16.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.474 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:18 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'8, mlcod=25'5)", 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:58.288753+0000", 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.182738782000001, 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037678569999999999, 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.288753+0000", 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.288753+0000", 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.288754+0000", 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4099999999999997e-07 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.288761+0000", 2024-09-16T03:40:16.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9439999999999999e-06 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.288772+0000", 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1318000000000001e-05 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.288790+0000", 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8026e-05 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.288834+0000", 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4180000000000001e-05 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.288850+0000", 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5500000000000001e-05 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.292486+0000", 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036363089999999999 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.292521+0000", 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4739000000000001e-05 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:20 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'10, mlcod=25'8)", 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:58.296291+0000", 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.175200799999999, 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00404033, 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.479 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296291+0000", 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296291+0000", 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296291+0000", 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0500000000000003e-07 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296373+0000", 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1823999999999995e-05 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296386+0000", 2024-09-16T03:40:16.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2747e-05 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296398+0000", 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2218999999999999e-05 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296456+0000", 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7361999999999997e-05 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296471+0000", 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5829999999999999e-05 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.300291+0000", 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038192090000000001 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.300331+0000", 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0433999999999999e-05 2024-09-16T03:40:16.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:21 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'11, mlcod=25'8)", 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:58.296742+0000", 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.174749069000001, 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038191739999999998, 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296742+0000", 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296742+0000", 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296743+0000", 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1800000000000005e-07 2024-09-16T03:40:16.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296807+0000", 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3683000000000007e-05 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296819+0000", 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2398000000000001e-05 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296824+0000", 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1000000000000003e-06 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296865+0000", 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1026999999999999e-05 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296878+0000", 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2754e-05 2024-09-16T03:40:16.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.300526+0000", 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036477929999999999 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.300562+0000", 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5701000000000003e-05 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:39 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'25, mlcod=25'22)", 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.153535375000001, 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042816670000000003, 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317957+0000", 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.54e-07 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318024+0000", 2024-09-16T03:40:16.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6730000000000007e-05 2024-09-16T03:40:16.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318033+0000", 2024-09-16T03:40:16.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1589999999999999e-06 2024-09-16T03:40:16.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318038+0000", 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8040000000000001e-06 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318080+0000", 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2475000000000002e-05 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318093+0000", 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3315e-05 2024-09-16T03:40:16.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.322203+0000", 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041094620000000004 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.322238+0000", 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4968000000000001e-05 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:121 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'83, mlcod=25'78)", 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:58.728887+0000", 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.742604533000002, 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039229859999999998, 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.728887+0000", 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.728887+0000", 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.728887+0000", 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9200000000000001e-07 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.728943+0000", 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5126e-05 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.728949+0000", 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7309999999999996e-06 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.728954+0000", 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1320000000000002e-06 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.728991+0000", 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6479000000000001e-05 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.729001+0000", 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0567000000000001e-05 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.732779+0000", 2024-09-16T03:40:16.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003777744 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.732810+0000", 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0715e-05 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:164 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'118, mlcod=25'113)", 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.31325824, 2024-09-16T03:40:16.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047753090000000002, 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158234+0000", 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8100000000000003e-07 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158294+0000", 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0359999999999998e-05 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158301+0000", 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3159999999999999e-06 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158305+0000", 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3900000000000002e-06 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158343+0000", 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7793000000000002e-05 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158361+0000", 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8230999999999999e-05 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.162979+0000", 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046178230000000001 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.163009+0000", 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9915000000000001e-05 2024-09-16T03:40:16.502 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:165 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'119, mlcod=25'114)", 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.159012+0000", 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.312479875000001, 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040153990000000002, 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.159012+0000", 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.159012+0000", 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.159012+0000", 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3099999999999997e-07 2024-09-16T03:40:16.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.159017+0000", 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4789999999999998e-06 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.159027+0000", 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8889999999999997e-06 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.159032+0000", 2024-09-16T03:40:16.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.451e-06 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.160072+0000", 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001040006 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.160084+0000", 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1518000000000001e-05 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.163010+0000", 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002926508 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.163027+0000", 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6917000000000001e-05 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:178 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'128, mlcod=25'125)", 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:16.507 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.187853376, 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058842529999999999, 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:16.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283639+0000", 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283643+0000", 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3980000000000001e-06 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283652+0000", 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6430000000000001e-06 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283656+0000", 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7339999999999999e-06 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283698+0000", 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1785000000000001e-05 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283715+0000", 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6555e-05 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289498+0000", 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0057832869999999998 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289522+0000", 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4290999999999998e-05 2024-09-16T03:40:16.511 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:179 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'129, mlcod=25'125)", 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.187382246999999, 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066372109999999996, 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284110+0000", 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.89e-07 2024-09-16T03:40:16.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284175+0000", 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5667000000000007e-05 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284186+0000", 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.009e-05 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284190+0000", 2024-09-16T03:40:16.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7600000000000002e-06 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284222+0000", 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1888999999999999e-05 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284234+0000", 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.225e-05 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290722+0000", 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064871909999999998 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290746+0000", 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4875000000000002e-05 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:180 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'130, mlcod=25'125)", 2024-09-16T03:40:16.516 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.186767937999999, 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072760109999999998, 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.68e-07 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284729+0000", 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4009999999999999e-06 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284738+0000", 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6940000000000007e-06 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284743+0000", 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5549999999999996e-06 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286547+0000", 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0018044770000000001 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286567+0000", 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9272000000000001e-05 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.291958+0000", 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053911540000000004 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292000+0000", 2024-09-16T03:40:16.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.189e-05 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:181 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'131, mlcod=25'126)", 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.186227387999999, 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067648710000000004, 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285265+0000", 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6100000000000001e-07 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285269+0000", 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5020000000000003e-06 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285279+0000", 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6879999999999994e-06 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285286+0000", 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4590000000000001e-06 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286842+0000", 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015560000000000001 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286854+0000", 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1959e-05 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292001+0000", 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051468859999999998 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292029+0000", 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7815999999999999e-05 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.525 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:182 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'132, mlcod=25'126)", 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.185782668000002, 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0063484680000000003, 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2200000000000004e-07 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285737+0000", 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7886999999999999e-05 2024-09-16T03:40:16.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285747+0000", 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9459999999999993e-06 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285752+0000", 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287024+0000", 2024-09-16T03:40:16.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012715409999999999 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287042+0000", 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8919999999999998e-05 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292031+0000", 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049888179999999999 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292057+0000", 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6057000000000001e-05 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:183 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'133, mlcod=25'126)", 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.185087574000001, 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060351210000000001, 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.530 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286405+0000", 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4300000000000001e-06 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286429+0000", 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3459999999999999e-05 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286439+0000", 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0105999999999999e-05 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286443+0000", 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.527e-06 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289328+0000", 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028841940000000001 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289340+0000", 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1861e-05 2024-09-16T03:40:16.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292411+0000", 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030717399999999999 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292439+0000", 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7803000000000001e-05 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:184 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'134, mlcod=25'126)", 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.184673206999999, 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056342509999999998, 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286819+0000", 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5499999999999998e-07 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286823+0000", 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2889999999999999e-06 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286833+0000", 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4769999999999992e-06 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286839+0000", 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6599999999999998e-06 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289572+0000", 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002733161 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289593+0000", 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0494999999999999e-05 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292440+0000", 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028470660000000001 2024-09-16T03:40:16.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292453+0000", 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2548000000000001e-05 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:185 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'135, mlcod=25'126)", 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.184109455000002, 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050824039999999996, 2024-09-16T03:40:16.539 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287383+0000", 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287410+0000", 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6985e-05 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287419+0000", 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6879999999999994e-06 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287424+0000", 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6829999999999999e-06 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289884+0000", 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002459639 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289917+0000", 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3303999999999998e-05 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292453+0000", 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025360769999999999 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292464+0000", 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1364e-05 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.543 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:186 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'136, mlcod=25'126)", 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.183594823, 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045805179999999996, 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3200000000000005e-07 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287902+0000", 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.634e-06 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287912+0000", 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7739999999999999e-06 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287916+0000", 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6099999999999999e-06 2024-09-16T03:40:16.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290274+0000", 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023574519999999999 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290289+0000", 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5481e-05 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292466+0000", 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021765439999999999 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292477+0000", 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:187 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'137, mlcod=25'126)", 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.288323+0000", 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.183168383999998, 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041669410000000004, 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288323+0000", 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288323+0000", 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288324+0000", 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3300000000000002e-07 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288328+0000", 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4689999999999999e-06 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288338+0000", 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8169999999999994e-06 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288343+0000", 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9370000000000003e-06 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290524+0000", 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021806009999999999 2024-09-16T03:40:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290536+0000", 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2751e-05 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292478+0000", 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001941454 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292490+0000", 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2279000000000001e-05 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:204 1.0 e26/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'150, mlcod=25'145)", 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.060226653000001, 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051027959999999997, 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.553 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:16.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411272+0000", 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1029999999999998e-06 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411280+0000", 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3129999999999999e-06 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411287+0000", 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2570000000000001e-06 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411347+0000", 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0214000000000003e-05 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411361+0000", 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3735e-05 2024-09-16T03:40:16.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416329+0000", 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049679889999999999 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416368+0000", 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8634999999999999e-05 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:205 1.0 e26/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'151, mlcod=26'147)", 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.059687385, 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004582226, 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411805+0000", 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2699999999999999e-07 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411808+0000", 2024-09-16T03:40:16.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6710000000000001e-06 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411815+0000", 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9480000000000002e-06 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411820+0000", 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0270000000000003e-06 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411851+0000", 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0474e-05 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411862+0000", 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.171e-05 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416370+0000", 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045072810000000001 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416386+0000", 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6588000000000001e-05 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:206 1.0 e26/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'152, mlcod=26'147)", 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:00.412162+0000", 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.059329553000001, 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042383129999999996, 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.562 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412162+0000", 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412162+0000", 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412162+0000", 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9299999999999998e-07 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412240+0000", 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7395e-05 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412247+0000", 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.294e-06 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412252+0000", 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8600000000000001e-06 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412285+0000", 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2843999999999999e-05 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412298+0000", 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3159000000000001e-05 2024-09-16T03:40:16.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416387+0000", 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040889890000000003 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416400+0000", 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3278999999999999e-05 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.567 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.567 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:16.567 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:40:16.569 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:16.565+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:16.570 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:76: done (0 left) 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:77: read oid 38 snap -1 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:77: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 23 current snap is 2 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:78: seq_num 61 ranges {376832=655360,1032192=376832} 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:78: writing smithi12138290-23 from 376832 to 1032192 tid 1 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:78: writing smithi12138290-23 from 1032192 to 1409024 tid 2 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:79: delete oid 43 current snap is 2 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:77: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:78: finishing write tid 1 to smithi12138290-23 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:78: finishing write tid 2 to smithi12138290-23 2024-09-16T03:40:16.572 INFO:tasks.rados.rados.0.smithi121.stdout:78: finishing write tid 3 to smithi12138290-23 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:78: finishing write tid 4 to smithi12138290-23 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:78: oid 23 updating version 0 to 16 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:78: oid 23 version 16 is already newer than 15 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:78: oid 23 version 16 is already newer than 14 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 23 v 16 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:78: left oid 23 (ObjNum 61 snap 2 seq_num 61) 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:77: done (2 left) 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:78: done (1 left) 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:79: done (0 left) 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:80: read oid 34 snap -1 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:80: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 48 current snap is 2 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:81: seq_num 63 ranges {802816=688128,1490944=376832} 2024-09-16T03:40:16.573 INFO:tasks.rados.rados.0.smithi121.stdout:81: writing smithi12138290-48 from 802816 to 1490944 tid 1 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:81: writing smithi12138290-48 from 1490944 to 1867776 tid 2 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:82: rollback oid 46 current snap is 2 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 46 to 1 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:80: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:82: finishing rollback tid 0 to smithi12138290-46 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:81: finishing write tid 1 to smithi12138290-48 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:81: finishing write tid 2 to smithi12138290-48 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:82: finishing rollback tid 1 to smithi12138290-46 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:81: finishing write tid 3 to smithi12138290-48 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:82: finishing rollback tid 2 to smithi12138290-46 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:81: finishing write tid 4 to smithi12138290-48 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:81: oid 48 updating version 0 to 13 2024-09-16T03:40:16.574 INFO:tasks.rados.rados.0.smithi121.stdout:81: oid 48 updating version 13 to 14 2024-09-16T03:40:16.575 INFO:tasks.rados.rados.0.smithi121.stdout:81: oid 48 version 14 is already newer than 12 2024-09-16T03:40:16.575 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 48 v 14 (ObjNum 63 snap 2 seq_num 63) dirty exists 2024-09-16T03:40:16.575 INFO:tasks.rados.rados.0.smithi121.stdout:81: left oid 48 (ObjNum 63 snap 2 seq_num 63) 2024-09-16T03:40:16.575 INFO:tasks.rados.rados.0.smithi121.stdout:80: done (2 left) 2024-09-16T03:40:16.575 INFO:tasks.rados.rados.0.smithi121.stdout:81: done (1 left) 2024-09-16T03:40:16.575 INFO:tasks.rados.rados.0.smithi121.stdout:82: done (0 left) 2024-09-16T03:40:16.575 INFO:tasks.rados.rados.0.smithi121.stdout:83: delete oid 49 current snap is 2 2024-09-16T03:40:16.576 INFO:tasks.rados.rados.0.smithi121.stdout:83: done (0 left) 2024-09-16T03:40:16.576 INFO:tasks.rados.rados.0.smithi121.stdout:84: read oid 10 snap 1 2024-09-16T03:40:16.576 INFO:tasks.rados.rados.0.smithi121.stdout:84: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:40:16.577 INFO:tasks.rados.rados.0.smithi121.stdout:85: snap_create 2024-09-16T03:40:16.578 INFO:tasks.rados.rados.0.smithi121.stdout:84: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:40:16.669 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:16.666+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:16.678 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:16.678 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:16.678 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:16.678 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:16.687 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:40:16.770 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:16.768+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:16.790 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:16.790 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:16.790 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:16.790 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:16.790 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:16.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:40:16.871 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:16.869+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:16.903 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:16.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:16.903 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:16.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.903 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.cs4 37/31 ECSubWrite(tid=60, reqid=client.4615.0:209, at_version=37'22, trim_to=0'0, roll_forward_to=34'20))", 2024-09-16T03:40:16.903 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.526059+0000", 2024-09-16T03:40:16.903 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.37450770300000003, 2024-09-16T03:40:16.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030832289999999998, 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.526059+0000", 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.526059+0000", 2024-09-16T03:40:16.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.526059+0000", 2024-09-16T03:40:16.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6600000000000003e-07 2024-09-16T03:40:16.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.526098+0000", 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8621000000000002e-05 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.526109+0000", 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0740000000000001e-05 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.526119+0000", 2024-09-16T03:40:16.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8039999999999996e-06 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.526189+0000", 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0763e-05 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.526200+0000", 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0316000000000001e-05 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529106+0000", 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029064759999999999 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529142+0000", 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6242999999999998e-05 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.cs4 37/31 ECSubWrite(tid=61, reqid=client.4615.0:209, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.529681+0000", 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.37088555400000001, 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015451200000000001, 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529681+0000", 2024-09-16T03:40:16.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529681+0000", 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529682+0000", 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7400000000000003e-07 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529691+0000", 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529701+0000", 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.764e-06 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529708+0000", 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4789999999999999e-06 2024-09-16T03:40:16.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529756+0000", 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7744999999999998e-05 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529765+0000", 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6330000000000002e-06 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:16.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.531204+0000", 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014392420000000001 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.531226+0000", 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2333000000000001e-05 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.913 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:210 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~557056,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.543835+0000", 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.356732098, 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002652522, 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 210 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543835+0000", 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543835+0000", 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543836+0000", 2024-09-16T03:40:16.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2079999999999999e-06 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543841+0000", 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9019999999999998e-06 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543852+0000", 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1501999999999999e-05 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543858+0000", 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8309999999999997e-06 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543908+0000", 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9699e-05 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543970+0000", 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1686000000000005e-05 2024-09-16T03:40:16.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546487+0000", 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002517694 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:211 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.543872+0000", 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.35669508599999999, 2024-09-16T03:40:16.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030446560000000002, 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 211 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543872+0000", 2024-09-16T03:40:16.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543872+0000", 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543872+0000", 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543874+0000", 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0499999999999999e-06 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543879+0000", 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1320000000000002e-06 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543882+0000", 2024-09-16T03:40:16.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9900000000000002e-06 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544421+0000", 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053922300000000002 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544507+0000", 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5419000000000003e-05 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546916+0000", 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024094659999999999 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:212 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.543899+0000", 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.35666826499999998, 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034416260000000001, 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.923 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 212 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543899+0000", 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543899+0000", 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543899+0000", 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0800000000000001e-07 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543901+0000", 2024-09-16T03:40:16.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3800000000000001e-06 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543905+0000", 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0419999999999996e-06 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.543910+0000", 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5650000000000003e-06 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545532+0000", 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016222070000000001 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545619+0000", 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7286000000000004e-05 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.547340+0000", 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0017208379999999999 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.927 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpRead(3.7s0 37/31 ECSubRead(tid=7, to_read={3:e441e0cd:::smithi12138290-34:head=0,139264,0}, subchunks={3:e441e0cd:::smithi12138290-34:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.544337+0000", 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.356229927, 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001160201, 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544337+0000", 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544337+0000", 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544337+0000", 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544337+0000", 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544360+0000", 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3492000000000001e-05 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544370+0000", 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.927e-06 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544820+0000", 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044923800000000002 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545497+0000", 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00067754399999999998 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.931 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpRead(3.7s0 37/31 ECSubRead(tid=8, to_read={3:e441e0cd:::smithi12138290-34:head=0,139264,0}, subchunks={3:e441e0cd:::smithi12138290-34:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.544734+0000", 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.35583283999999998, 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001168245, 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544734+0000", 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544734+0000", 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544734+0000", 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544734+0000", 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544779+0000", 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4796000000000003e-05 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544788+0000", 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9679999999999995e-06 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545815+0000", 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001026994 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545902+0000", 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7486999999999998e-05 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.935 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=7, attrs_read=0))", 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.544920+0000", 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.35564696299999998, 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001043188, 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544920+0000", 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544920+0000", 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544921+0000", 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9500000000000002e-07 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545000+0000", 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9711999999999998e-05 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545011+0000", 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0492000000000001e-05 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545017+0000", 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545926+0000", 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00090874300000000005 2024-09-16T03:40:16.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545963+0000", 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6820000000000003e-05 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=8, attrs_read=0))", 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.545034+0000", 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.355532819, 2024-09-16T03:40:16.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00098588499999999993, 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545034+0000", 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545034+0000", 2024-09-16T03:40:16.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545034+0000", 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7899999999999999e-07 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545124+0000", 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9468999999999993e-05 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545136+0000", 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1719000000000001e-05 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545141+0000", 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.524e-06 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545993+0000", 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00085160499999999996 2024-09-16T03:40:16.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546020+0000", 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189e-05 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=7, attrs_read=0))", 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.545051+0000", 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.35551591599999999, 2024-09-16T03:40:16.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011142249999999999, 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545051+0000", 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545051+0000", 2024-09-16T03:40:16.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545052+0000", 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9699999999999995e-07 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545259+0000", 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000207674 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545270+0000", 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0566e-05 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545276+0000", 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2009999999999996e-06 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546091+0000", 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000814713 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546165+0000", 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4374000000000003e-05 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=7, attrs_read=0))", 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.545485+0000", 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.35508145400000002, 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001016248, 2024-09-16T03:40:16.947 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545485+0000", 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545485+0000", 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545485+0000", 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545485+0000", 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545503+0000", 2024-09-16T03:40:16.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7479999999999999e-05 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545511+0000", 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.585e-06 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546221+0000", 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070946099999999999 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546502+0000", 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00028072200000000001 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=8, attrs_read=0))", 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.545891+0000", 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.354675342, 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0010481780000000001, 2024-09-16T03:40:16.951 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545891+0000", 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545891+0000", 2024-09-16T03:40:16.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545891+0000", 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545891+0000", 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545904+0000", 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3036e-05 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545911+0000", 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7309999999999996e-06 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546638+0000", 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00072646899999999999 2024-09-16T03:40:16.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546940+0000", 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00030194199999999998 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=9, attrs_read=0))", 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.545978+0000", 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.354588401, 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0010281590000000001, 2024-09-16T03:40:16.955 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545978+0000", 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545978+0000", 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545979+0000", 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000002e-07 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546059+0000", 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0187999999999999e-05 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546070+0000", 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0251000000000001e-05 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546076+0000", 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2210000000000002e-06 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546979+0000", 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00090343599999999997 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.547007+0000", 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7288999999999999e-05 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=9, attrs_read=0))", 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.546023+0000", 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.35454332700000002, 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001025548, 2024-09-16T03:40:16.959 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546023+0000", 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546023+0000", 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546024+0000", 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3000000000000001e-07 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546127+0000", 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000102797 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546134+0000", 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9310000000000002e-06 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546139+0000", 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0900000000000004e-06 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.547029+0000", 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000890134 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.547049+0000", 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0066000000000001e-05 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=9, attrs_read=0))", 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.546040+0000", 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.35452634900000002, 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001045292, 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.963 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546040+0000", 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546040+0000", 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546041+0000", 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7499999999999999e-07 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546169+0000", 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000128078 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546180+0000", 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0528e-05 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.546184+0000", 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3000000000000003e-06 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.547067+0000", 2024-09-16T03:40:16.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00088326799999999999 2024-09-16T03:40:16.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.547086+0000", 2024-09-16T03:40:16.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8343e-05 2024-09-16T03:40:16.967 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.967 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.967 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es5 37/31 ECSubWrite(tid=26, reqid=client.4615.0:218, at_version=37'8, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.556607+0000", 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.34395963400000001, 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016395260000000001, 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556607+0000", 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556607+0000", 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556608+0000", 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3200000000000005e-07 2024-09-16T03:40:16.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556623+0000", 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5432999999999999e-05 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556636+0000", 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3198e-05 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556644+0000", 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9889999999999992e-06 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556770+0000", 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012522999999999999 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556779+0000", 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3149999999999998e-06 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558212+0000", 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014331070000000001 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558247+0000", 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721999999999997e-05 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.972 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es5 37/31 ECSubWrite(tid=27, reqid=client.4615.0:219, at_version=37'9, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.557444+0000", 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.34312324599999999, 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036476130000000001, 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557444+0000", 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557444+0000", 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557444+0000", 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0800000000000004e-07 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557474+0000", 2024-09-16T03:40:16.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9978999999999999e-05 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557486+0000", 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1338999999999999e-05 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557492+0000", 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5010000000000001e-06 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557569+0000", 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7101999999999995e-05 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557578+0000", 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9560000000000003e-06 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561055+0000", 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003476973 2024-09-16T03:40:16.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561091+0000", 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6055e-05 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es5 37/31 ECSubWrite(tid=28, reqid=client.4615.0:219, at_version=0'0, trim_to=0'0, roll_forward_to=37'9))", 2024-09-16T03:40:16.977 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.561476+0000", 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.339090803, 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022714240000000002, 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561476+0000", 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561476+0000", 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561477+0000", 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3499999999999996e-07 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561489+0000", 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2561999999999999e-05 2024-09-16T03:40:16.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561504+0000", 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4351e-05 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561513+0000", 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.499e-06 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561568+0000", 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5234e-05 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561577+0000", 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0440000000000001e-06 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.563712+0000", 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021348209999999999 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.563747+0000", 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5277999999999997e-05 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpRead(3.0s1 37/31 ECSubRead(tid=17, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.573658+0000", 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.32690837099999998, 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026375309999999998, 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.982 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573658+0000", 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573658+0000", 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573659+0000", 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2599999999999999e-07 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573663+0000", 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1300000000000003e-06 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573673+0000", 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0694e-05 2024-09-16T03:40:16.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573679+0000", 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0619999999999997e-06 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573730+0000", 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0813999999999998e-05 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.576296+0000", 2024-09-16T03:40:16.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025657050000000002 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpRead(3.0s1 37/31 ECSubRead(tid=18, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.573916+0000", 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.32665043300000002, 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002504661, 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:16.986 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573916+0000", 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573916+0000", 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573917+0000", 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2599999999999998e-07 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573922+0000", 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8470000000000003e-06 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573930+0000", 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5350000000000004e-06 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573935+0000", 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2719999999999997e-06 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.576335+0000", 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023994139999999999 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.576421+0000", 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6167000000000004e-05 2024-09-16T03:40:16.989 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.990 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.990 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.990 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:16.990 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:16.990 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:16.991 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:40:16.992 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:16.990+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:17.092 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:17.090+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:17.095 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:17.095 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:17.095 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:17.095 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:17.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:40:17.194 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:17.191+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:17.207 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:17.207 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:17.207 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:17.207 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:17.208 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:17.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:40:17.294 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:17.293+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:17.320 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:17.320 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:17.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:17.320 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:17.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.320 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpRead(3.7s1 37/31 ECSubRead(tid=7, to_read={3:e441e0cd:::smithi12138290-34:head=0,139264,0}, subchunks={3:e441e0cd:::smithi12138290-34:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.544413+0000", 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.773597592, 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071498199999999997, 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544413+0000", 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544413+0000", 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544413+0000", 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4700000000000001e-07 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544420+0000", 2024-09-16T03:40:17.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7889999999999997e-06 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544435+0000", 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5107e-05 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544445+0000", 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8709999999999992e-06 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544520+0000", 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5362000000000001e-05 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545128+0000", 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060770599999999996 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.324 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpRead(3.7s1 37/31 ECSubRead(tid=8, to_read={3:e441e0cd:::smithi12138290-34:head=0,139264,0}, subchunks={3:e441e0cd:::smithi12138290-34:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.544783+0000", 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.77322733499999996, 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061253900000000005, 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544783+0000", 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544783+0000", 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544784+0000", 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4000000000000001e-07 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544793+0000", 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1900000000000001e-06 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544809+0000", 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6685999999999999e-05 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.544819+0000", 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8060000000000006e-06 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545168+0000", 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00034860099999999998 2024-09-16T03:40:17.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545395+0000", 2024-09-16T03:40:17.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022751600000000001 2024-09-16T03:40:17.328 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.328 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.328 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpRead(3.7s1 37/31 ECSubRead(tid=9, to_read={3:e441e0cd:::smithi12138290-34:head=0,139264,0}, subchunks={3:e441e0cd:::smithi12138290-34:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.545790+0000", 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.772220291, 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015468799999999999, 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545790+0000", 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545790+0000", 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545791+0000", 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1900000000000001e-07 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545798+0000", 2024-09-16T03:40:17.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1300000000000003e-06 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545808+0000", 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9939999999999995e-06 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545814+0000", 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0669999999999997e-06 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545866+0000", 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2117999999999998e-05 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.545945+0000", 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8559999999999993e-05 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:217 3.es0 3:7acde04a:::smithi12138290-46:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.554771+0000", 2024-09-16T03:40:17.332 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76323902899999996, 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037061720000000002, 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 217 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554771+0000", 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554771+0000", 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554773+0000", 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5579999999999999e-06 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554779+0000", 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9440000000000002e-06 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554791+0000", 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2699e-05 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554798+0000", 2024-09-16T03:40:17.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2920000000000001e-06 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554847+0000", 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9814000000000001e-05 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554910+0000", 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2887999999999998e-05 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.555426+0000", 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051603200000000004 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557667+0000", 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022401399999999998 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558408+0000", 2024-09-16T03:40:17.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00074108699999999997 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558477+0000", 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9717999999999999e-05 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:218 3.es0 3:7acde04a:::smithi12138290-46:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.554812+0000", 2024-09-16T03:40:17.338 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76319843099999995, 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064025410000000003, 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 218 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554812+0000", 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554812+0000", 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554812+0000", 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554815+0000", 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1200000000000002e-06 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554822+0000", 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8179999999999998e-06 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554827+0000", 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.344e-06 2024-09-16T03:40:17.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.555949+0000", 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001121597 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556045+0000", 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5501000000000006e-05 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556494+0000", 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044957900000000003 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558578+0000", 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020843419999999999 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561164+0000", 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025858840000000001 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561214+0000", 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9913999999999997e-05 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:219 3.es0 3:7acde04a:::smithi12138290-46:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.554835+0000", 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76317492099999995, 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065377689999999997, 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.344 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 219 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554835+0000", 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554835+0000", 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554836+0000", 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8000000000000001e-07 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554838+0000", 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0209999999999998e-06 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554843+0000", 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7830000000000004e-06 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.554848+0000", 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7140000000000001e-06 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556859+0000", 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020107509999999999 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556949+0000", 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9988000000000006e-05 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557400+0000", 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00045093100000000001 2024-09-16T03:40:17.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561230+0000", 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038298609999999999 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561329+0000", 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9220000000000002e-05 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561373+0000", 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4119999999999998e-05 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=25, last_complete=37'7, committed=1, applied=1))", 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.556381+0000", 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76162909499999998, 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001232894, 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.350 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556381+0000", 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556381+0000", 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556385+0000", 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3250000000000001e-06 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556390+0000", 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0540000000000002e-06 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556401+0000", 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0828e-05 2024-09-16T03:40:17.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556408+0000", 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.182e-06 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557601+0000", 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011920819999999999 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557614+0000", 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3423e-05 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=25, last_complete=37'7, committed=1, applied=1))", 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.556468+0000", 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76154203399999998, 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001173552, 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.354 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556468+0000", 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556468+0000", 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556469+0000", 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0800000000000004e-07 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556473+0000", 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0080000000000004e-06 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556487+0000", 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4693e-05 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556503+0000", 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5756999999999998e-05 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557632+0000", 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001129213 2024-09-16T03:40:17.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557642+0000", 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2730000000000008e-06 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=25, last_complete=37'7, committed=1, applied=1))", 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.556528+0000", 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76148176499999998, 2024-09-16T03:40:17.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001139078, 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556528+0000", 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556528+0000", 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556529+0000", 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4600000000000004e-07 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556532+0000", 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1769999999999998e-06 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556540+0000", 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6939999999999995e-06 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556553+0000", 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3145999999999999e-05 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557658+0000", 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001105347 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557667+0000", 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0680000000000003e-06 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=25, last_complete=37'7, committed=1, applied=1))", 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.556804+0000", 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76120639999999995, 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00088534699999999998, 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.362 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556804+0000", 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556804+0000", 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556804+0000", 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7399999999999998e-07 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556807+0000", 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.52e-06 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556813+0000", 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5130000000000002e-06 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556822+0000", 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1519999999999998e-06 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557682+0000", 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00085911400000000001 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557689+0000", 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5739999999999998e-06 2024-09-16T03:40:17.365 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=25, last_complete=37'7, committed=1, applied=1))", 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.556955+0000", 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76105561700000002, 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075804799999999999, 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556955+0000", 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556955+0000", 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556955+0000", 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8099999999999999e-07 2024-09-16T03:40:17.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556957+0000", 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089999999999999e-06 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556963+0000", 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6999999999999996e-06 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556968+0000", 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5509999999999999e-06 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557705+0000", 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073693599999999999 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557713+0000", 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3710000000000002e-06 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.369 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=26, last_complete=37'8, committed=1, applied=1))", 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.557134+0000", 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76087615099999995, 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060217799999999998, 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557134+0000", 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557134+0000", 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557135+0000", 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4099999999999999e-07 2024-09-16T03:40:17.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557138+0000", 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4539999999999999e-06 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557147+0000", 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1339999999999993e-06 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557154+0000", 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1230000000000002e-06 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557729+0000", 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00057430100000000004 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557736+0000", 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6249999999999998e-06 2024-09-16T03:40:17.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=26, last_complete=37'8, committed=1, applied=1))", 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.557511+0000", 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76049933400000003, 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024352, 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.374 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557511+0000", 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557511+0000", 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557511+0000", 2024-09-16T03:40:17.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6899999999999998e-07 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557515+0000", 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5470000000000001e-06 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557524+0000", 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8929999999999994e-06 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557532+0000", 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0050000000000004e-06 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557747+0000", 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000215708 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557754+0000", 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8979999999999999e-06 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.377 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=25, last_complete=37'7, committed=1, applied=1))", 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.558258+0000", 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.75975268200000001, 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023133100000000001, 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558258+0000", 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558258+0000", 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558258+0000", 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5099999999999999e-07 2024-09-16T03:40:17.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558262+0000", 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8630000000000001e-06 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558272+0000", 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9660000000000008e-06 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558279+0000", 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5400000000000001e-06 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558334+0000", 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5362000000000003e-05 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558489+0000", 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015494900000000001 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.381 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=26, last_complete=37'8, committed=1, applied=1))", 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.558285+0000", 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.75972546699999999, 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024373400000000001, 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558285+0000", 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558285+0000", 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558285+0000", 2024-09-16T03:40:17.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1600000000000001e-07 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558289+0000", 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0790000000000002e-06 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558300+0000", 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.023e-05 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558306+0000", 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558512+0000", 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020567599999999999 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558528+0000", 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6688e-05 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=27, last_complete=37'9, committed=1, applied=1))", 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.560985+0000", 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.75702543799999999, 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103379, 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560985+0000", 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560985+0000", 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560985+0000", 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2200000000000004e-07 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560989+0000", 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8970000000000001e-06 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560999+0000", 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8749999999999995e-06 2024-09-16T03:40:17.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561005+0000", 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2559999999999999e-06 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561070+0000", 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.478e-05 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561088+0000", 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7949e-05 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=26, last_complete=37'8, committed=1, applied=1))", 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.561016+0000", 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.75699417000000002, 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021491899999999999, 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.390 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561016+0000", 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561016+0000", 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561017+0000", 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3e-07 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561021+0000", 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4399999999999998e-06 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561032+0000", 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0604e-05 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561038+0000", 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7059999999999998e-06 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561110+0000", 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1952000000000005e-05 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561231+0000", 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000120487 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=27, last_complete=37'9, committed=1, applied=1))", 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.561070+0000", 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.75693987900000004, 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000201404, 2024-09-16T03:40:17.394 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561070+0000", 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561070+0000", 2024-09-16T03:40:17.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561071+0000", 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7e-07 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561074+0000", 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9529999999999999e-06 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561082+0000", 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6500000000000002e-06 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561096+0000", 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3243e-05 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561258+0000", 2024-09-16T03:40:17.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016264099999999999 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561272+0000", 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3546999999999999e-05 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=27, last_complete=37'9, committed=1, applied=1))", 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.561129+0000", 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.75688161700000001, 2024-09-16T03:40:17.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058683900000000002, 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561129+0000", 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561129+0000", 2024-09-16T03:40:17.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561129+0000", 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2699999999999999e-07 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561133+0000", 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.89e-06 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561142+0000", 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7520000000000002e-06 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561148+0000", 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6640000000000003e-06 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561291+0000", 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000142943 2024-09-16T03:40:17.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561715+0000", 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042486299999999999 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=28, last_complete=37'9, committed=1, applied=1))", 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.563718+0000", 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.754292199, 2024-09-16T03:40:17.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8119000000000001e-05, 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.563718+0000", 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.563718+0000", 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:17.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.563719+0000", 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5400000000000001e-07 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.563723+0000", 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1509999999999997e-06 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.563734+0000", 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1637000000000001e-05 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.563742+0000", 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9619999999999993e-06 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.563795+0000", 2024-09-16T03:40:17.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2955999999999999e-05 2024-09-16T03:40:17.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:17.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:17.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:17.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.563806+0000", 2024-09-16T03:40:17.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0759e-05 2024-09-16T03:40:17.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.408 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:17.408 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:17.409 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:17.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:40:17.411 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:17.408+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:17.511 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:17.509+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T03:40:17.527 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:17.527 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:17.527 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:17.527 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:17.532 INFO:tasks.rados.rados.0.smithi121.stdout:84: done (1 left) 2024-09-16T03:40:17.532 INFO:tasks.rados.rados.0.smithi121.stdout:85: done (0 left) 2024-09-16T03:40:17.532 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 24 current snap is 3 2024-09-16T03:40:17.533 INFO:tasks.rados.rados.0.smithi121.stdout:86: seq_num 65 ranges {524288=720896,1245184=376832} 2024-09-16T03:40:17.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:40:17.539 INFO:tasks.rados.rados.0.smithi121.stdout:86: writing smithi12138290-24 from 524288 to 1245184 tid 1 2024-09-16T03:40:17.541 INFO:tasks.rados.rados.0.smithi121.stdout:86: writing smithi12138290-24 from 1245184 to 1622016 tid 2 2024-09-16T03:40:17.541 INFO:tasks.rados.rados.0.smithi121.stdout:87: copy_from oid 4 from oid 46 current snap is 3 2024-09-16T03:40:17.541 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 34 current snap is 3 2024-09-16T03:40:17.541 INFO:tasks.rados.rados.0.smithi121.stdout:88: seq_num 66 ranges {557056=737280,1294336=376832} 2024-09-16T03:40:17.548 INFO:tasks.rados.rados.0.smithi121.stdout:88: writing smithi12138290-34 from 557056 to 1294336 tid 1 2024-09-16T03:40:17.551 INFO:tasks.rados.rados.0.smithi121.stdout:88: writing smithi12138290-34 from 1294336 to 1671168 tid 2 2024-09-16T03:40:17.551 INFO:tasks.rados.rados.0.smithi121.stdout:89: snap_create 2024-09-16T03:40:17.551 INFO:tasks.rados.rados.0.smithi121.stdout:86: finishing write tid 1 to smithi12138290-24 2024-09-16T03:40:17.551 INFO:tasks.rados.rados.0.smithi121.stdout:86: finishing write tid 2 to smithi12138290-24 2024-09-16T03:40:17.551 INFO:tasks.rados.rados.0.smithi121.stdout:86: finishing write tid 3 to smithi12138290-24 2024-09-16T03:40:17.553 INFO:tasks.rados.rados.0.smithi121.stdout:86: finishing write tid 4 to smithi12138290-24 2024-09-16T03:40:17.553 INFO:tasks.rados.rados.0.smithi121.stdout:86: oid 24 updating version 0 to 9 2024-09-16T03:40:17.553 INFO:tasks.rados.rados.0.smithi121.stdout:86: oid 24 updating version 9 to 11 2024-09-16T03:40:17.553 INFO:tasks.rados.rados.0.smithi121.stdout:86: oid 24 version 11 is already newer than 10 2024-09-16T03:40:17.553 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 24 v 11 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-09-16T03:40:17.553 INFO:tasks.rados.rados.0.smithi121.stdout:86: left oid 24 (ObjNum 65 snap 3 seq_num 65) 2024-09-16T03:40:17.555 INFO:tasks.rados.rados.0.smithi121.stdout:87: finishing copy_from racing read to smithi12138290-4 2024-09-16T03:40:17.556 INFO:tasks.rados.rados.0.smithi121.stdout:88: finishing write tid 1 to smithi12138290-34 2024-09-16T03:40:17.556 INFO:tasks.rados.rados.0.smithi121.stdout:88: finishing write tid 2 to smithi12138290-34 2024-09-16T03:40:17.557 INFO:tasks.rados.rados.0.smithi121.stdout:88: finishing write tid 3 to smithi12138290-34 2024-09-16T03:40:17.557 INFO:tasks.rados.rados.0.smithi121.stdout:87: finishing copy_from to smithi12138290-4 2024-09-16T03:40:17.558 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 4 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:40:17.559 INFO:tasks.rados.rados.0.smithi121.stdout:88: finishing write tid 4 to smithi12138290-34 2024-09-16T03:40:17.559 INFO:tasks.rados.rados.0.smithi121.stdout:88: oid 34 updating version 0 to 8 2024-09-16T03:40:17.559 INFO:tasks.rados.rados.0.smithi121.stdout:88: oid 34 version 8 is already newer than 6 2024-09-16T03:40:17.559 INFO:tasks.rados.rados.0.smithi121.stdout:88: oid 34 version 8 is already newer than 7 2024-09-16T03:40:17.559 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 34 v 8 (ObjNum 66 snap 3 seq_num 66) dirty exists 2024-09-16T03:40:17.559 INFO:tasks.rados.rados.0.smithi121.stdout:88: left oid 34 (ObjNum 66 snap 3 seq_num 66) 2024-09-16T03:40:17.611 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:17.610+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:17.641 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:17.641 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:17.642 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:17.642 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:17.642 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:17.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:40:17.713 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:17.711+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:17.754 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 37/31 ECSubWrite(tid=60, reqid=client.4615.0:209, at_version=37'22, trim_to=0'0, roll_forward_to=34'20))", 2024-09-16T03:40:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.526020+0000", 2024-09-16T03:40:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.2259904319999999, 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001541396, 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526020+0000", 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526020+0000", 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526020+0000", 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7000000000000001e-07 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526048+0000", 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8382999999999999e-05 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526061+0000", 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3047000000000001e-05 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526073+0000", 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1742e-05 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526150+0000", 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7155000000000002e-05 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526164+0000", 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3661e-05 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527519+0000", 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001355476 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527561+0000", 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1662000000000002e-05 2024-09-16T03:40:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 37/31 ECSubWrite(tid=61, reqid=client.4615.0:209, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.529518+0000", 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.222491687, 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0007471, 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529518+0000", 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529518+0000", 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529519+0000", 2024-09-16T03:40:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.850000000000001e-07 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529534+0000", 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4316e-05 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529551+0000", 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7722000000000001e-05 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529560+0000", 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4719999999999995e-06 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529627+0000", 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6983000000000006e-05 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529633+0000", 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0700000000000003e-06 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530225+0000", 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00059259899999999997 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530265+0000", 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9953000000000002e-05 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=25, reqid=client.4615.0:217, at_version=37'7, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.555333+0000", 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.196676625, 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010782770000000001, 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555333+0000", 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555333+0000", 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555334+0000", 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4299999999999999e-07 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555358+0000", 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4924999999999999e-05 2024-09-16T03:40:17.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555367+0000", 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4619999999999996e-06 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555374+0000", 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4390000000000003e-06 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555443+0000", 2024-09-16T03:40:17.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8359000000000001e-05 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555452+0000", 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1989999999999995e-06 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556371+0000", 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000918642 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556412+0000", 2024-09-16T03:40:17.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1007999999999998e-05 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=26, reqid=client.4615.0:218, at_version=37'8, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.556485+0000", 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.1955245329999999, 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000870234, 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556485+0000", 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556485+0000", 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556486+0000", 2024-09-16T03:40:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3300000000000001e-07 2024-09-16T03:40:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556511+0000", 2024-09-16T03:40:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4722000000000001e-05 2024-09-16T03:40:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556520+0000", 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3589999999999997e-06 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556528+0000", 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556597+0000", 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9097999999999995e-05 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556610+0000", 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2847000000000001e-05 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557316+0000", 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00070549000000000002 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557356+0000", 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9860999999999997e-05 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=27, reqid=client.4615.0:219, at_version=37'9, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.557379+0000", 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.1946308130000001, 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002658929, 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557379+0000", 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557379+0000", 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557380+0000", 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6100000000000001e-07 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557395+0000", 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5763000000000001e-05 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557408+0000", 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2183e-05 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557415+0000", 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.622e-06 2024-09-16T03:40:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557481+0000", 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5617999999999994e-05 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557490+0000", 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0280000000000006e-06 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.560006+0000", 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025159779999999999 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.560038+0000", 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2175999999999998e-05 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=28, reqid=client.4615.0:219, at_version=0'0, trim_to=0'0, roll_forward_to=37'9))", 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.561444+0000", 2024-09-16T03:40:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.1905660949999999, 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00071886499999999996, 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561444+0000", 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561444+0000", 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561445+0000", 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1699999999999997e-07 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561456+0000", 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1408e-05 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561471+0000", 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5284000000000001e-05 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561480+0000", 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7620000000000001e-06 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561543+0000", 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2725999999999995e-05 2024-09-16T03:40:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561552+0000", 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8570000000000001e-06 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562126+0000", 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00057423800000000003 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562163+0000", 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6872999999999997e-05 2024-09-16T03:40:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 37/31 ECSubWrite(tid=14, reqid=client.4615.0:220, at_version=37'10, trim_to=0'0, roll_forward_to=34'8))", 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.571470+0000", 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.180539617, 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00087942399999999996, 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571470+0000", 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571470+0000", 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571470+0000", 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.08e-07 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571485+0000", 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4426999999999999e-05 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571495+0000", 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571501+0000", 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7610000000000004e-06 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571554+0000", 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2268999999999997e-05 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571559+0000", 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.699e-06 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.572323+0000", 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076385799999999996 2024-09-16T03:40:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.572350+0000", 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6684000000000001e-05 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 37/31 ECSubWrite(tid=15, reqid=client.4615.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.573070+0000", 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.1789401879999999, 2024-09-16T03:40:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020277009999999998, 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573070+0000", 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573070+0000", 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573070+0000", 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573078+0000", 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1919999999999999e-06 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573105+0000", 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7830999999999998e-05 2024-09-16T03:40:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573112+0000", 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6499999999999999e-06 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573189+0000", 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7373999999999995e-05 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573196+0000", 2024-09-16T03:40:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.575065+0000", 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001869419 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.575097+0000", 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2014999999999998e-05 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 38/31 ECSubWrite(tid=62, reqid=client.4615.0:225, at_version=38'9, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.542570+0000", 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.20944030999999999, 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018344209999999999, 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.542570+0000", 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.542570+0000", 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.542570+0000", 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4000000000000001e-07 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.543309+0000", 2024-09-16T03:40:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073942400000000003 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.543321+0000", 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1773999999999999e-05 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.543330+0000", 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.543392+0000", 2024-09-16T03:40:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2198000000000003e-05 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.543399+0000", 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4490000000000002e-06 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544371+0000", 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00097108700000000004 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544404+0000", 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3552e-05 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 38/31 ECSubWrite(tid=63, reqid=client.4615.0:226, at_version=38'10, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.544027+0000", 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.207983162, 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001061338, 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544027+0000", 2024-09-16T03:40:17.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544027+0000", 2024-09-16T03:40:17.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544028+0000", 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1000000000000001e-06 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544393+0000", 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00036505000000000001 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544401+0000", 2024-09-16T03:40:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3489999999999992e-06 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544409+0000", 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7749999999999993e-06 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544447+0000", 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8164000000000003e-05 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544453+0000", 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4009999999999996e-06 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.545066+0000", 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00061321499999999996 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.545088+0000", 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2283999999999999e-05 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 38/31 ECSubWrite(tid=64, reqid=client.4615.0:227, at_version=38'11, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.544740+0000", 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.20727020099999999, 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00054331099999999999, 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544740+0000", 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544740+0000", 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544740+0000", 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0500000000000003e-07 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544750+0000", 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0064999999999999e-05 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544758+0000", 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.216e-06 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544764+0000", 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0730000000000001e-06 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544812+0000", 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8111000000000001e-05 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544818+0000", 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4119999999999998e-06 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.545267+0000", 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044835799999999999 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.545283+0000", 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6470999999999999e-05 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 38/31 ECSubWrite(tid=65, reqid=client.4615.0:227, at_version=0'0, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.550357+0000", 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.20165344900000001, 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047359700000000003, 2024-09-16T03:40:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.550357+0000", 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.550357+0000", 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.550357+0000", 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.61e-07 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.550366+0000", 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5699999999999993e-06 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.550377+0000", 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0956e-05 2024-09-16T03:40:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.550382+0000", 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2259999999999996e-06 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.550428+0000", 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6533999999999999e-05 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.550435+0000", 2024-09-16T03:40:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1650000000000003e-06 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.550806+0000", 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037167500000000002 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.550830+0000", 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3810000000000001e-05 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 38/31 ECSubWrite(tid=20, reqid=client.4615.0:229, at_version=38'11, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.553469+0000", 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.19854122900000001, 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023386560000000002, 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553469+0000", 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553469+0000", 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553469+0000", 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.01e-07 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553994+0000", 2024-09-16T03:40:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00052489500000000005 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554009+0000", 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4907e-05 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554020+0000", 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0691e-05 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554091+0000", 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1778000000000003e-05 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554098+0000", 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.331e-06 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555769+0000", 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001671723 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555807+0000", 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7929999999999998e-05 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 38/31 ECSubWrite(tid=21, reqid=client.4615.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.557279+0000", 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.194731139, 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00053224999999999995, 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557279+0000", 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557279+0000", 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557280+0000", 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1799999999999999e-06 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557292+0000", 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1673e-05 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557302+0000", 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0786e-05 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557308+0000", 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7069999999999997e-06 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557360+0000", 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1663000000000003e-05 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557369+0000", 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1050000000000001e-06 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557779+0000", 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00041055099999999999 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557811+0000", 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1585000000000003e-05 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:17.817 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:17.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:40:17.820 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:17.817+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:17.920 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:17.918+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:17.922 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:17.922 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:17.922 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:17.922 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:17.931 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:40:18.022 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:18.020+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:18.035 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:18.035 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:18.035 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:18.035 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:18.035 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:18.037 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:18.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:40:18.123 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:18.120+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:18.147 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs6 37/31 ECSubWrite(tid=60, reqid=client.4615.0:209, at_version=37'22, trim_to=0'0, roll_forward_to=34'20))", 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.526004+0000", 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.6196917420000001, 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001691632, 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526004+0000", 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526004+0000", 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526005+0000", 2024-09-16T03:40:18.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4900000000000002e-07 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526045+0000", 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0373000000000001e-05 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526063+0000", 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7866000000000001e-05 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526073+0000", 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0502e-05 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526174+0000", 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010061899999999999 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526184+0000", 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7189999999999996e-06 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527658+0000", 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014739550000000001 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527696+0000", 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8349000000000002e-05 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs6 37/31 ECSubWrite(tid=61, reqid=client.4615.0:209, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T03:40:18.152 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.529513+0000", 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.616182539, 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00066609999999999998, 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529513+0000", 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529513+0000", 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529514+0000", 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0999999999999997e-07 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529524+0000", 2024-09-16T03:40:18.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9869999999999994e-06 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529536+0000", 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1409999999999999e-05 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529542+0000", 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3770000000000001e-06 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529600+0000", 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7476999999999999e-05 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.529610+0000", 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0755e-05 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530143+0000", 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00053265699999999997 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530180+0000", 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6526999999999998e-05 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.bs1 37/31 ECSubWrite(tid=26, reqid=client.4615.0:213, at_version=37'12, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.554715+0000", 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.5909805480000001, 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018993529999999999, 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.157 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.554715+0000", 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.554715+0000", 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.554716+0000", 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1199999999999999e-07 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555017+0000", 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00030086000000000002 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555036+0000", 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8909000000000001e-05 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555047+0000", 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1127999999999999e-05 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555134+0000", 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7724000000000002e-05 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555146+0000", 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2103000000000001e-05 2024-09-16T03:40:18.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556566+0000", 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014196409999999999 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556615+0000", 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8675999999999999e-05 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.bs1 37/31 ECSubWrite(tid=27, reqid=client.4615.0:214, at_version=37'13, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.556867+0000", 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.588829391, 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001256191, 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556867+0000", 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556867+0000", 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556868+0000", 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.186e-06 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556918+0000", 2024-09-16T03:40:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0127000000000003e-05 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556928+0000", 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7350000000000008e-06 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556935+0000", 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4449999999999999e-06 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557002+0000", 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6588999999999996e-05 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557012+0000", 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9150000000000008e-06 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557951+0000", 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00093958300000000002 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558123+0000", 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017161100000000001 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.bs1 37/31 ECSubWrite(tid=28, reqid=client.4615.0:215, at_version=37'14, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.558021+0000", 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.587675387, 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012697590000000001, 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.166 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558021+0000", 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558021+0000", 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558021+0000", 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1200000000000002e-07 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558041+0000", 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9491999999999999e-05 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558061+0000", 2024-09-16T03:40:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0248999999999999e-05 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558067+0000", 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.19e-06 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558166+0000", 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8746000000000006e-05 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558175+0000", 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1470000000000007e-06 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.559264+0000", 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001089046 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.559290+0000", 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6077000000000001e-05 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.bs1 37/31 ECSubWrite(tid=29, reqid=client.4615.0:215, at_version=0'0, trim_to=0'0, roll_forward_to=37'14))", 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.561420+0000", 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.584275766, 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080123399999999995, 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.171 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561420+0000", 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561420+0000", 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561421+0000", 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.23e-07 2024-09-16T03:40:18.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561432+0000", 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0567999999999999e-05 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561444+0000", 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2326e-05 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561450+0000", 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0750000000000003e-06 2024-09-16T03:40:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561559+0000", 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000108899 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561568+0000", 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4269999999999997e-06 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562081+0000", 2024-09-16T03:40:18.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00051276000000000002 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562221+0000", 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014045600000000001 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpRead(3.bs1 37/31 ECSubRead(tid=30, to_read={3:de0fee80:::smithi12138290-48:head=0,4096,0}, subchunks={3:de0fee80:::smithi12138290-48:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.562091+0000", 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.5836045080000001, 2024-09-16T03:40:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000279719, 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562091+0000", 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562091+0000", 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562092+0000", 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9599999999999998e-07 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562102+0000", 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7739999999999999e-06 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562116+0000", 2024-09-16T03:40:18.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4185000000000001e-05 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562126+0000", 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.628e-06 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562279+0000", 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015291 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.178 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562371+0000", 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2425999999999999e-05 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.7s5 38/31 ECSubWrite(tid=10, reqid=client.4615.0:231, at_version=38'6, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.552058+0000", 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.59363798599999995, 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017036009999999999, 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.179 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.552058+0000", 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.552058+0000", 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.552058+0000", 2024-09-16T03:40:18.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7700000000000001e-07 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.552521+0000", 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00046324300000000001 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.552532+0000", 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0604999999999999e-05 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.552538+0000", 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2230000000000004e-06 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.552588+0000", 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9951000000000003e-05 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.552594+0000", 2024-09-16T03:40:18.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1700000000000002e-06 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553727+0000", 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001132391 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553762+0000", 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4841000000000001e-05 2024-09-16T03:40:18.183 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s1 38/31 ECSubWrite(tid=20, reqid=client.4615.0:229, at_version=38'11, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.553635+0000", 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.59206149699999999, 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020639230000000001, 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.184 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553635+0000", 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553635+0000", 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553635+0000", 2024-09-16T03:40:18.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.22e-07 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554412+0000", 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00077744200000000004 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554421+0000", 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2380000000000003e-06 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554427+0000", 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2000000000000002e-06 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554481+0000", 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4038e-05 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554486+0000", 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7930000000000003e-06 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555670+0000", 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011833200000000001 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555698+0000", 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8670000000000002e-05 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.188 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.7s5 38/31 ECSubWrite(tid=11, reqid=client.4615.0:232, at_version=38'7, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.553944+0000", 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.59175175300000005, 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0016518030000000001, 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553944+0000", 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553944+0000", 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553945+0000", 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3499999999999996e-07 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554136+0000", 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00019075200000000001 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554146+0000", 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0793e-05 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554152+0000", 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2809999999999999e-06 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554200+0000", 2024-09-16T03:40:18.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8656999999999998e-05 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554209+0000", 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555557+0000", 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001348084 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555596+0000", 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8986000000000002e-05 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.7s5 38/31 ECSubWrite(tid=12, reqid=client.4615.0:233, at_version=38'8, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.555264+0000", 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.59043179499999998, 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00075214300000000004, 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.193 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555264+0000", 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555264+0000", 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555265+0000", 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2099999999999996e-07 2024-09-16T03:40:18.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555282+0000", 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7320999999999999e-05 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555296+0000", 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4090999999999999e-05 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555309+0000", 2024-09-16T03:40:18.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2943e-05 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555380+0000", 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0797000000000007e-05 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555397+0000", 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6864999999999999e-05 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555991+0000", 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00059457200000000003 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.556016+0000", 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4933e-05 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.197 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.7s5 38/31 ECSubWrite(tid=13, reqid=client.4615.0:233, at_version=0'0, trim_to=0'0, roll_forward_to=38'8))", 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.557081+0000", 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.58861452700000005, 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00046963300000000001, 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557081+0000", 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557081+0000", 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557082+0000", 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2099999999999997e-07 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557089+0000", 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1679999999999998e-06 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557098+0000", 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2029999999999998e-06 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557104+0000", 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1090000000000003e-06 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557144+0000", 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9598000000000003e-05 2024-09-16T03:40:18.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557150+0000", 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7729999999999996e-06 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557520+0000", 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037018899999999999 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557551+0000", 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1072000000000003e-05 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s1 38/31 ECSubWrite(tid=21, reqid=client.4615.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.557307+0000", 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.58838875599999996, 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00062788999999999998, 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:18.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557307+0000", 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557307+0000", 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557308+0000", 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1399999999999996e-07 2024-09-16T03:40:18.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557319+0000", 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1296e-05 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557333+0000", 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3664e-05 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557341+0000", 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6109999999999996e-06 2024-09-16T03:40:18.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557417+0000", 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6798000000000006e-05 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557427+0000", 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4979999999999995e-06 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557903+0000", 2024-09-16T03:40:18.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047601400000000003 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.557935+0000", 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2295000000000002e-05 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:18.206 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:18.207 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:18.208 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:18.208 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:18.209 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:18.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:40:18.210 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:18.210 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:18.219 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:18.219 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:18.223 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:18.221+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:18.235 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:18.235 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:18.252 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:18.269 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:18.269 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:18.286 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:18.287 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:18.302 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:18.302 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:18.319 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:18.319 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:18.324 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:18.321+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:18.336 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:18.336 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:18.353 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:18.353 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:18.354 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:18.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:40:18.369 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:18.386 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:18.386 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:18.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:18.403 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:18.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:18.401+0000 7efc921c4640 0 -- 172.21.15.76:0/3202124093 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7efc8c220c80 msgr2=0x7efc8c223070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:18.446 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:18.443+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:18.505 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:18.505 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:18.514 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:40:18.527 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:18.525+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:18.534 INFO:tasks.rados.rados.0.smithi121.stdout:86: done (3 left) 2024-09-16T03:40:18.534 INFO:tasks.rados.rados.0.smithi121.stdout:87: done (2 left) 2024-09-16T03:40:18.534 INFO:tasks.rados.rados.0.smithi121.stdout:88: done (1 left) 2024-09-16T03:40:18.534 INFO:tasks.rados.rados.0.smithi121.stdout:89: done (0 left) 2024-09-16T03:40:18.534 INFO:tasks.rados.rados.0.smithi121.stdout:90: snap_create 2024-09-16T03:40:18.621 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:18.621 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:18.628 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:18.626+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:18.658 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:18.658 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:18.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:40:18.728 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:18.725+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:18.770 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:18.770 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:18.779 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:40:18.830 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:18.828+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:18.922 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:18.922 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:18.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:40:18.931 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:18.928+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:19.031 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:19.030+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:19.034 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:19.034 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:19.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:40:19.133 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:19.130+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:19.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:40:19.234 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:19.232+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:19.271 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:19.271 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:19.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:40:19.363 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:19.362+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:19.423 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:19.423 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:19.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:40:19.436 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:19.434+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:19.537 INFO:tasks.rados.rados.0.smithi121.stdout:90: done (0 left) 2024-09-16T03:40:19.537 INFO:tasks.rados.rados.0.smithi121.stdout:91: snap_remove snap 4 2024-09-16T03:40:19.538 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:19.536+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:19.541 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:40:19.576 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:19.577 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:19.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:40:19.639 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:19.637+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:19.687 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:19.687 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:19.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:40:19.739 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:19.738+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:19.798 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:19.798 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:19.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:40:19.840 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:19.838+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:19.908 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:19.909 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:19.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:40:19.941 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:19.939+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:20.019 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:20.019 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:20.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:40:20.042 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:20.040+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:20.130 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:20.130 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:20.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:40:20.142 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:20.141+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:20.240 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:20.240 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:20.243 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:20.240+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:20.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:40:20.343 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:20.342+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:20.353 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:20.353 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:20.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:40:20.444 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:20.443+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:20.504 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:20.504 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:20.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:40:20.528 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:40:20.528 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:40:20.528 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T03:40:20.528 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T03:40:20.534 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T03:40:20.537 INFO:tasks.rados.rados.0.smithi121.stdout:91: done (0 left) 2024-09-16T03:40:20.537 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 33 current snap is 5 2024-09-16T03:40:20.537 INFO:tasks.rados.rados.0.smithi121.stdout:92: seq_num 67 ranges {540672=753664,1294336=376832} 2024-09-16T03:40:20.544 INFO:tasks.rados.rados.0.smithi121.stdout:92: writing smithi12138290-33 from 540672 to 1294336 tid 1 2024-09-16T03:40:20.544 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:20.543+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:20.546 INFO:tasks.rados.rados.0.smithi121.stdout:92: writing smithi12138290-33 from 1294336 to 1671168 tid 2 2024-09-16T03:40:20.547 INFO:tasks.rados.rados.0.smithi121.stdout:93: read oid 44 snap -1 2024-09-16T03:40:20.547 INFO:tasks.rados.rados.0.smithi121.stdout:93: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-16T03:40:20.547 INFO:tasks.rados.rados.0.smithi121.stdout:94: delete oid 20 current snap is 5 2024-09-16T03:40:20.556 INFO:tasks.rados.rados.0.smithi121.stdout:94: done (2 left) 2024-09-16T03:40:20.556 INFO:tasks.rados.rados.0.smithi121.stdout:95: copy_from oid 43 from oid 9 current snap is 5 2024-09-16T03:40:20.556 INFO:tasks.rados.rados.0.smithi121.stdout:96: copy_from oid 11 from oid 41 current snap is 5 2024-09-16T03:40:20.556 INFO:tasks.rados.rados.0.smithi121.stdout:97: snap_remove snap 2 2024-09-16T03:40:20.632 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:20.632 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:20.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:20.641 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:40:20.646 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:20.645+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:20.746 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:20.745+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:20.761 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:20.761 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:20.779 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:20.779 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:20.799 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:20.799 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:20.800 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:20.812 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:40:20.812 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:20.813 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:20.829 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:20.829 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 41, 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "ecf7a42a-e941-4b24-8693-0ff8b6eb41b1", 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T03:39:42.686123+0000", 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T03:40:20.531501+0000", 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2024-09-16T03:39:56.235015+0000", 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2024-09-16T03:39:46.749797+0000", 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5801984, 2024-09-16T03:40:20.839 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "noscrub", 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 7, 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 3, 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 16, 2024-09-16T03:40:20.840 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "jewel", 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "squid", 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_crimson": false, 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".mgr", 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T03:39:56.952065+0000", 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T03:40:20.841 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "is_stretch_pool": false, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2024-09-16T03:40:20.842 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "26", 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:40:20.843 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T03:40:20.844 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_max": 32, 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2024-09-16T03:40:20.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": {} 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "read_balance": { 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "score_type": "Fair distribution", 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "score_acting": 15.380000114440918, 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "score_stable": 15.380000114440918, 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "optimal_score": 0.12999999523162842, 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "raw_score_acting": 2, 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "raw_score_stable": 2, 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity_weighted": 1, 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "average_primary_affinity": 1, 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: "average_primary_affinity_weighted": 1 2024-09-16T03:40:20.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "rbd", 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T03:39:57.213209+0000", 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T03:40:20.848 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:40:20.849 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:40:20.849 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:40:20.849 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:40:20.849 INFO:teuthology.orchestra.run.smithi076.stdout: "is_stretch_pool": false, 2024-09-16T03:40:20.849 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T03:40:20.849 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:40:20.849 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 8, 2024-09-16T03:40:20.849 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 8, 2024-09-16T03:40:20.849 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 8, 2024-09-16T03:40:20.849 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 8, 2024-09-16T03:40:20.850 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 8, 2024-09-16T03:40:20.850 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T03:40:20.850 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T03:40:20.850 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T03:40:20.850 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T03:40:20.850 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T03:40:20.850 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T03:40:20.850 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T03:40:20.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.850 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "27", 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 2, 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 27, 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T03:40:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T03:40:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T03:40:20.853 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "read_balance": { 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "score_type": "Fair distribution", 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "score_acting": 6.0199999809265137, 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "score_stable": 6.0199999809265137, 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "optimal_score": 0.81000000238418579, 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "raw_score_acting": 4.880000114440918, 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "raw_score_stable": 4.880000114440918, 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity_weighted": 1, 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "average_primary_affinity": 1, 2024-09-16T03:40:20.854 INFO:teuthology.orchestra.run.smithi076.stdout: "average_primary_affinity_weighted": 1 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T03:40:09.501089+0000", 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 3, 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 7, 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 5, 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 1, 2024-09-16T03:40:20.855 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "is_stretch_pool": false, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T03:40:20.856 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "41", 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T03:40:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 6, 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 41, 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T03:40:20.858 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "shecprofile", 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T03:40:20.859 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 16384, 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.860 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "8820e1f9-b619-44a5-abb1-7cfa97fcef00", 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.861 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:20.863 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:20.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2024-09-16T03:40:20.865 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2834882614 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/2834882614", 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/2834882614", 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/2834882614", 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/2834882614", 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.866 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "0f58ae45-e2ff-4c4f-9875-7d34ec312cbe", 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 22, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.867 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6808", 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6809", 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.868 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6810", 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6811", 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.869 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6814", 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6815", 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.870 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6812", 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6813", 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2045143946 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.871 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.103:6809/2045143946", 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.103:6811/2045143946", 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.103:6815/2045143946", 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.103:6813/2045143946", 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b369fe92-86b8-45b3-a418-024f0e4becd7", 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.872 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 0, 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.873 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6816", 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6817", 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.874 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6818", 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6819", 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.875 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6822", 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6823", 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6820", 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6821", 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 273257286 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.877 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.119:6817/273257286", 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.119:6819/273257286", 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.119:6823/273257286", 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.119:6821/273257286", 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b9521bf1-46ef-402c-8472-c1b476280b23", 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.878 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 0, 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6809", 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:20.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6811", 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.880 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6813", 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6815", 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6821", 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6823", 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.882 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6817", 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6819", 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 278480145 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.121:6811/278480145", 2024-09-16T03:40:20.883 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.121:6815/278480145", 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.121:6823/278480145", 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.121:6819/278480145", 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "15bec6a1-fc13-4822-8cfa-899b83886c10", 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.884 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 0, 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.886 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.887 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.888 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3302785479 2024-09-16T03:40:20.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/3302785479", 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/3302785479", 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/3302785479", 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/3302785479", 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.890 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "e9b234ec-bfa4-4f72-adab-295e067b2e61", 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 0, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6800", 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6801", 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6802", 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6803", 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.893 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6806", 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6807", 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6804", 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6805", 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3765230361 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.103:6801/3765230361", 2024-09-16T03:40:20.895 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.103:6803/3765230361", 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.103:6807/3765230361", 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.103:6805/3765230361", 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "6d4ae9fb-8d8f-451d-a12f-1265302b03ab", 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.896 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 22, 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6824", 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:20.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6825", 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6826", 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:20.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6827", 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6830", 2024-09-16T03:40:20.899 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6831", 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.900 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6828", 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6829", 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 446165121 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.119:6825/446165121", 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.119:6827/446165121", 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.119:6831/446165121", 2024-09-16T03:40:20.901 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.119:6829/446165121", 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "2c9456ed-00ec-46aa-814e-f85eb441fc4f", 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.902 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 0, 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6808", 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6810", 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6812", 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:20.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6814", 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6820", 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:20.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6822", 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6816", 2024-09-16T03:40:20.906 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:20.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.907 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.907 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6818", 2024-09-16T03:40:20.907 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1304791608 2024-09-16T03:40:20.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.907 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.907 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.121:6810/1304791608", 2024-09-16T03:40:20.907 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.121:6814/1304791608", 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.121:6822/1304791608", 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.121:6818/1304791608", 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 8, 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b618a02a-8628-48f5-bb5f-f7591aafa118", 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.908 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:20.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2024-09-16T03:40:20.910 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.911 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.912 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2630853906 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/2630853906", 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/2630853906", 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/2630853906", 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/2630853906", 2024-09-16T03:40:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 9, 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "dc3e5be7-f012-47e6-ba35-0e3354ca0c18", 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.914 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6816", 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6817", 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6818", 2024-09-16T03:40:20.916 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6819", 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.917 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6822", 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6823", 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6820", 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6821", 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2702548675 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.103:6817/2702548675", 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.103:6819/2702548675", 2024-09-16T03:40:20.919 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.103:6823/2702548675", 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.103:6821/2702548675", 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 10, 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "4e70a7a8-7ba8-42ab-a45a-e0564b561619", 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.920 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 22, 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6808", 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:20.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6809", 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6810", 2024-09-16T03:40:20.922 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6811", 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6814", 2024-09-16T03:40:20.923 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6815", 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.924 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6812", 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6813", 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 205142394 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.119:6809/205142394", 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.119:6811/205142394", 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.119:6815/205142394", 2024-09-16T03:40:20.925 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.119:6813/205142394", 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 11, 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "d16bc493-335f-41c2-bd68-cbad93c49a5d", 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.926 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6824", 2024-09-16T03:40:20.927 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6825", 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6826", 2024-09-16T03:40:20.928 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6827", 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.929 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6830", 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6831", 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.930 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6828", 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6829", 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1551091359 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.121:6825/1551091359", 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.121:6827/1551091359", 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.121:6831/1551091359", 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.121:6829/1551091359", 2024-09-16T03:40:20.931 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 12, 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1", 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.932 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.933 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:20.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 890992293 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/890992293", 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/890992293", 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/890992293", 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/890992293", 2024-09-16T03:40:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 13, 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "61e9204b-8269-4c35-b409-5cfbd5e6abbb", 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.938 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6824", 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6825", 2024-09-16T03:40:20.939 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6826", 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.940 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6827", 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6830", 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.941 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6831", 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6828", 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.103:6829", 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3905392668 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.103:6825/3905392668", 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.103:6827/3905392668", 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.103:6831/3905392668", 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.103:6829/3905392668", 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.943 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 14, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "0bd5338e-eb99-44cc-9f5f-1c8ef903d02c", 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0.5100555419921875, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 21, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.944 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6800", 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6801", 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.945 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6802", 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6803", 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6806", 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6807", 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.947 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6804", 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.119:6805", 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1147733023 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.948 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.119:6801/1147733023", 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.119:6803/1147733023", 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.119:6807/1147733023", 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.119:6805/1147733023", 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 15, 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f240405e-d268-41cc-bd99-5a2ca71a85a9", 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T03:40:20.949 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 23, 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 31, 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6800", 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6801", 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6802", 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6803", 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6806", 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.952 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6807", 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6804", 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.953 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.121:6805", 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3149764080 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.121:6801/3149764080", 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.121:6803/3149764080", 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3149764080", 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.121:6805/3149764080", 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:53.946705+0000", 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.955 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:53.541012+0000", 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.956 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.279946+0000", 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.202293+0000", 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:53.783644+0000", 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.098081+0000", 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.959 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.053567+0000", 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.960 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.262433+0000", 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 8, 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.271589+0000", 2024-09-16T03:40:20.961 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 9, 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.040223+0000", 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 10, 2024-09-16T03:40:20.962 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.048985+0000", 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 11, 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.963 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.049910+0000", 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 12, 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.019981+0000", 2024-09-16T03:40:20.964 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 13, 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:53.956380+0000", 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 14, 2024-09-16T03:40:20.965 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:53.072636+0000", 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 15, 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540701547738038271, 2024-09-16T03:40:20.966 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:54.047013+0000", 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [], 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [], 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_primaries": [], 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": {}, 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "range_blocklist": {}, 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2024-09-16T03:40:20.967 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "shecprofile": { 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "c": "2", 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-device-class": "", 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-num-failure-domains": "0", 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-osds-per-failure-domain": "0", 2024-09-16T03:40:20.968 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-root": "default", 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "4", 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "3", 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "shecprofile", 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "shec", 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "multiple" 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 2, 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2024-09-16T03:40:20.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 6, 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [ 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.970 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 6, 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2024-09-16T03:40:20.971 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2024-09-16T03:40:20.972 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.972 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:20.972 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:20.973 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:20.975 INFO:tasks.thrashosds.thrasher:j is {'epoch': 41, 'fsid': 'ecf7a42a-e941-4b24-8693-0ff8b6eb41b1', 'created': '2024-09-16T03:39:42.686123+0000', 'modified': '2024-09-16T03:40:20.531501+0000', 'last_up_change': '2024-09-16T03:39:56.235015+0000', 'last_in_change': '2024-09-16T03:39:46.749797+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T03:39:56.952065+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T03:39:57.213209+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 27, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.019999980926514, 'score_stable': 6.019999980926514, 'optimal_score': 0.8100000023841858, 'raw_score_acting': 4.880000114440918, 'raw_score_stable': 4.880000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T03:40:09.501089+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 41, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '8820e1f9-b619-44a5-abb1-7cfa97fcef00', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 2834882614}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 2834882614}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 2834882614}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 2834882614}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 2834882614}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 2834882614}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 2834882614}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 2834882614}]}, 'public_addr': '172.21.15.76:6811/2834882614', 'cluster_addr': '172.21.15.76:6813/2834882614', 'heartbeat_back_addr': '172.21.15.76:6817/2834882614', 'heartbeat_front_addr': '172.21.15.76:6815/2834882614', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0f58ae45-e2ff-4c4f-9875-7d34ec312cbe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6808', 'nonce': 2045143946}, {'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 2045143946}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6810', 'nonce': 2045143946}, {'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 2045143946}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6814', 'nonce': 2045143946}, {'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 2045143946}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6812', 'nonce': 2045143946}, {'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 2045143946}]}, 'public_addr': '172.21.15.103:6809/2045143946', 'cluster_addr': '172.21.15.103:6811/2045143946', 'heartbeat_back_addr': '172.21.15.103:6815/2045143946', 'heartbeat_front_addr': '172.21.15.103:6813/2045143946', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b369fe92-86b8-45b3-a418-024f0e4becd7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6816', 'nonce': 273257286}, {'type': 'v1', 'addr': '172.21.15.119:6817', 'nonce': 273257286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6818', 'nonce': 273257286}, {'type': 'v1', 'addr': '172.21.15.119:6819', 'nonce': 273257286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6822', 'nonce': 273257286}, {'type': 'v1', 'addr': '172.21.15.119:6823', 'nonce': 273257286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6820', 'nonce': 273257286}, {'type': 'v1', 'addr': '172.21.15.119:6821', 'nonce': 273257286}]}, 'public_addr': '172.21.15.119:6817/273257286', 'cluster_addr': '172.21.15.119:6819/273257286', 'heartbeat_back_addr': '172.21.15.119:6823/273257286', 'heartbeat_front_addr': '172.21.15.119:6821/273257286', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'b9521bf1-46ef-402c-8472-c1b476280b23', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6809', 'nonce': 278480145}, {'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 278480145}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6813', 'nonce': 278480145}, {'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 278480145}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6821', 'nonce': 278480145}, {'type': 'v1', 'addr': '172.21.15.121:6823', 'nonce': 278480145}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6817', 'nonce': 278480145}, {'type': 'v1', 'addr': '172.21.15.121:6819', 'nonce': 278480145}]}, 'public_addr': '172.21.15.121:6811/278480145', 'cluster_addr': '172.21.15.121:6815/278480145', 'heartbeat_back_addr': '172.21.15.121:6823/278480145', 'heartbeat_front_addr': '172.21.15.121:6819/278480145', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '15bec6a1-fc13-4822-8cfa-899b83886c10', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 3302785479}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 3302785479}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 3302785479}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 3302785479}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 3302785479}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 3302785479}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 3302785479}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 3302785479}]}, 'public_addr': '172.21.15.76:6803/3302785479', 'cluster_addr': '172.21.15.76:6805/3302785479', 'heartbeat_back_addr': '172.21.15.76:6809/3302785479', 'heartbeat_front_addr': '172.21.15.76:6807/3302785479', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e9b234ec-bfa4-4f72-adab-295e067b2e61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6800', 'nonce': 3765230361}, {'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 3765230361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6802', 'nonce': 3765230361}, {'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 3765230361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6806', 'nonce': 3765230361}, {'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 3765230361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6804', 'nonce': 3765230361}, {'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 3765230361}]}, 'public_addr': '172.21.15.103:6801/3765230361', 'cluster_addr': '172.21.15.103:6803/3765230361', 'heartbeat_back_addr': '172.21.15.103:6807/3765230361', 'heartbeat_front_addr': '172.21.15.103:6805/3765230361', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6d4ae9fb-8d8f-451d-a12f-1265302b03ab', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6824', 'nonce': 446165121}, {'type': 'v1', 'addr': '172.21.15.119:6825', 'nonce': 446165121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6826', 'nonce': 446165121}, {'type': 'v1', 'addr': '172.21.15.119:6827', 'nonce': 446165121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6830', 'nonce': 446165121}, {'type': 'v1', 'addr': '172.21.15.119:6831', 'nonce': 446165121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6828', 'nonce': 446165121}, {'type': 'v1', 'addr': '172.21.15.119:6829', 'nonce': 446165121}]}, 'public_addr': '172.21.15.119:6825/446165121', 'cluster_addr': '172.21.15.119:6827/446165121', 'heartbeat_back_addr': '172.21.15.119:6831/446165121', 'heartbeat_front_addr': '172.21.15.119:6829/446165121', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '2c9456ed-00ec-46aa-814e-f85eb441fc4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6808', 'nonce': 1304791608}, {'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1304791608}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6812', 'nonce': 1304791608}, {'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 1304791608}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6820', 'nonce': 1304791608}, {'type': 'v1', 'addr': '172.21.15.121:6822', 'nonce': 1304791608}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6816', 'nonce': 1304791608}, {'type': 'v1', 'addr': '172.21.15.121:6818', 'nonce': 1304791608}]}, 'public_addr': '172.21.15.121:6810/1304791608', 'cluster_addr': '172.21.15.121:6814/1304791608', 'heartbeat_back_addr': '172.21.15.121:6822/1304791608', 'heartbeat_front_addr': '172.21.15.121:6818/1304791608', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b618a02a-8628-48f5-bb5f-f7591aafa118', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 2630853906}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 2630853906}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 2630853906}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 2630853906}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 2630853906}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 2630853906}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 2630853906}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 2630853906}]}, 'public_addr': '172.21.15.76:6819/2630853906', 'cluster_addr': '172.21.15.76:6821/2630853906', 'heartbeat_back_addr': '172.21.15.76:6825/2630853906', 'heartbeat_front_addr': '172.21.15.76:6823/2630853906', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'dc3e5be7-f012-47e6-ba35-0e3354ca0c18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6816', 'nonce': 2702548675}, {'type': 'v1', 'addr': '172.21.15.103:6817', 'nonce': 2702548675}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6818', 'nonce': 2702548675}, {'type': 'v1', 'addr': '172.21.15.103:6819', 'nonce': 2702548675}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6822', 'nonce': 2702548675}, {'type': 'v1', 'addr': '172.21.15.103:6823', 'nonce': 2702548675}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6820', 'nonce': 2702548675}, {'type': 'v1', 'addr': '172.21.15.103:6821', 'nonce': 2702548675}]}, 'public_addr': '172.21.15.103:6817/2702548675', 'cluster_addr': '172.21.15.103:6819/2702548675', 'heartbeat_back_addr': '172.21.15.103:6823/2702548675', 'heartbeat_front_addr': '172.21.15.103:6821/2702548675', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '4e70a7a8-7ba8-42ab-a45a-e0564b561619', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6808', 'nonce': 205142394}, {'type': 'v1', 'addr': '172.21.15.119:6809', 'nonce': 205142394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6810', 'nonce': 205142394}, {'type': 'v1', 'addr': '172.21.15.119:6811', 'nonce': 205142394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6814', 'nonce': 205142394}, {'type': 'v1', 'addr': '172.21.15.119:6815', 'nonce': 205142394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6812', 'nonce': 205142394}, {'type': 'v1', 'addr': '172.21.15.119:6813', 'nonce': 205142394}]}, 'public_addr': '172.21.15.119:6809/205142394', 'cluster_addr': '172.21.15.119:6811/205142394', 'heartbeat_back_addr': '172.21.15.119:6815/205142394', 'heartbeat_front_addr': '172.21.15.119:6813/205142394', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'd16bc493-335f-41c2-bd68-cbad93c49a5d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6824', 'nonce': 1551091359}, {'type': 'v1', 'addr': '172.21.15.121:6825', 'nonce': 1551091359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6826', 'nonce': 1551091359}, {'type': 'v1', 'addr': '172.21.15.121:6827', 'nonce': 1551091359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6830', 'nonce': 1551091359}, {'type': 'v1', 'addr': '172.21.15.121:6831', 'nonce': 1551091359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6828', 'nonce': 1551091359}, {'type': 'v1', 'addr': '172.21.15.121:6829', 'nonce': 1551091359}]}, 'public_addr': '172.21.15.121:6825/1551091359', 'cluster_addr': '172.21.15.121:6827/1551091359', 'heartbeat_back_addr': '172.21.15.121:6831/1551091359', 'heartbeat_front_addr': '172.21.15.121:6829/1551091359', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 890992293}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 890992293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 890992293}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 890992293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 890992293}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 890992293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 890992293}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 890992293}]}, 'public_addr': '172.21.15.76:6827/890992293', 'cluster_addr': '172.21.15.76:6829/890992293', 'heartbeat_back_addr': '172.21.15.76:6833/890992293', 'heartbeat_front_addr': '172.21.15.76:6831/890992293', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '61e9204b-8269-4c35-b409-5cfbd5e6abbb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6824', 'nonce': 3905392668}, {'type': 'v1', 'addr': '172.21.15.103:6825', 'nonce': 3905392668}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6826', 'nonce': 3905392668}, {'type': 'v1', 'addr': '172.21.15.103:6827', 'nonce': 3905392668}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6830', 'nonce': 3905392668}, {'type': 'v1', 'addr': '172.21.15.103:6831', 'nonce': 3905392668}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6828', 'nonce': 3905392668}, {'type': 'v1', 'addr': '172.21.15.103:6829', 'nonce': 3905392668}]}, 'public_addr': '172.21.15.103:6825/3905392668', 'cluster_addr': '172.21.15.103:6827/3905392668', 'heartbeat_back_addr': '172.21.15.103:6831/3905392668', 'heartbeat_front_addr': '172.21.15.103:6829/3905392668', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '0bd5338e-eb99-44cc-9f5f-1c8ef903d02c', 'up': 1, 'in': 1, 'weight': 0.5100555419921875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6800', 'nonce': 1147733023}, {'type': 'v1', 'addr': '172.21.15.119:6801', 'nonce': 1147733023}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6802', 'nonce': 1147733023}, {'type': 'v1', 'addr': '172.21.15.119:6803', 'nonce': 1147733023}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6806', 'nonce': 1147733023}, {'type': 'v1', 'addr': '172.21.15.119:6807', 'nonce': 1147733023}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6804', 'nonce': 1147733023}, {'type': 'v1', 'addr': '172.21.15.119:6805', 'nonce': 1147733023}]}, 'public_addr': '172.21.15.119:6801/1147733023', 'cluster_addr': '172.21.15.119:6803/1147733023', 'heartbeat_back_addr': '172.21.15.119:6807/1147733023', 'heartbeat_front_addr': '172.21.15.119:6805/1147733023', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'f240405e-d268-41cc-bd99-5a2ca71a85a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6800', 'nonce': 3149764080}, {'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3149764080}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6802', 'nonce': 3149764080}, {'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3149764080}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6806', 'nonce': 3149764080}, {'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3149764080}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6804', 'nonce': 3149764080}, {'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3149764080}]}, 'public_addr': '172.21.15.121:6801/3149764080', 'cluster_addr': '172.21.15.121:6803/3149764080', 'heartbeat_back_addr': '172.21.15.121:6807/3149764080', 'heartbeat_front_addr': '172.21.15.121:6805/3149764080', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:53.946705+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:53.541012+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.279946+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.202293+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:53.783644+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.098081+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.053567+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.262433+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.271589+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.040223+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.048985+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.049910+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.019981+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:53.956380+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:53.072636+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:54.047013+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'shecprofile': {'c': '2', 'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'k': '4', 'm': '3', 'name': 'shecprofile', 'plugin': 'shec', 'technique': 'multiple'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 6, 'length': 1}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 6, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T03:40:20.975 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:20.975 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:20.976 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:20.976 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:20.976 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:20.977 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:20.977 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:20.977 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:20.978 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:20.979 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:20.971+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:20.979 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:202 3.cs0 3:335b675a:::smithi12138290-38:head [read 0~622592,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.517297+0000", 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4000297939999999, 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030985510000000002, 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:20.980 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 202 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517297+0000", 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:20.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517297+0000", 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517299+0000", 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.012e-06 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517309+0000", 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4069999999999999e-06 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517323+0000", 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3757000000000001e-05 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517333+0000", 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0183000000000001e-05 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517387+0000", 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4616e-05 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517393+0000", 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4600000000000002e-06 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.518733+0000", 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001339891 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.518789+0000", 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6307999999999997e-05 2024-09-16T03:40:20.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520396+0000", 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606917 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:203 3.cs0 3:335b675a:::smithi12138290-38:head [read 0~622592,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.517358+0000", 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3999692679999995, 2024-09-16T03:40:20.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035734669999999999, 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 203 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517358+0000", 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517358+0000", 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517358+0000", 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517363+0000", 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2089999999999999e-06 2024-09-16T03:40:20.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517368+0000", 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.361e-06 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517371+0000", 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4450000000000001e-06 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517449+0000", 2024-09-16T03:40:20.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7540999999999995e-05 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517455+0000", 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519211+0000", 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017551070000000001 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519297+0000", 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6359000000000004e-05 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520931+0000", 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016344980000000001 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:204 3.cs0 3:335b675a:::smithi12138290-38:head [read 0~622592,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:20.990 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.517379+0000", 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.399948126, 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047868290000000003, 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 204 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517379+0000", 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517379+0000", 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517379+0000", 2024-09-16T03:40:20.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1199999999999999e-07 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517381+0000", 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8789999999999999e-06 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517385+0000", 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.687e-06 2024-09-16T03:40:20.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517388+0000", 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9079999999999999e-06 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517480+0000", 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2122999999999999e-05 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517483+0000", 2024-09-16T03:40:20.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520973+0000", 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034891940000000001 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.521058+0000", 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5296999999999993e-05 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.522166+0000", 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011079499999999999 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/31 ECSubReadReply(tid=57, attrs_read=0))", 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.519486+0000", 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3978410849999996, 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00092706199999999998, 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:20.996 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519486+0000", 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519486+0000", 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519486+0000", 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.84e-07 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519908+0000", 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00042131299999999998 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519923+0000", 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5639000000000001e-05 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519928+0000", 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3250000000000001e-06 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520049+0000", 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121678 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520413+0000", 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00036382299999999999 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:20.999 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/31 ECSubReadReply(tid=58, attrs_read=0))", 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.519808+0000", 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.397519569, 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001136019, 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519808+0000", 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519808+0000", 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519808+0000", 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.84e-07 2024-09-16T03:40:21.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520179+0000", 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037134800000000002 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520187+0000", 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0080000000000002e-06 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520193+0000", 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1739999999999999e-06 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520608+0000", 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00041499100000000003 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520944+0000", 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033611399999999999 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.003 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 37/31 ECSubWrite(tid=23, reqid=client.4615.0:205, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.524999+0000", 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3923281120000004, 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014483740000000001, 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.524999+0000", 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.524999+0000", 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.524999+0000", 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4600000000000001e-07 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525283+0000", 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028382599999999999 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525294+0000", 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0754000000000001e-05 2024-09-16T03:40:21.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525299+0000", 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2800000000000003e-06 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525340+0000", 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0621000000000003e-05 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525348+0000", 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0169999999999996e-06 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526413+0000", 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00106529 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526447+0000", 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4440000000000002e-05 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.008 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:209 3.cs0 3:3e925a6e:::smithi12138290-43:head [delete] snapc 3=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.525370+0000", 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.391957143, 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040223389999999998, 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:21.009 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 209 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525370+0000", 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525370+0000", 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525371+0000", 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525374+0000", 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4709999999999999e-06 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525381+0000", 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0090000000000001e-06 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525386+0000", 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525429+0000", 2024-09-16T03:40:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2889000000000002e-05 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525480+0000", 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1023999999999998e-05 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525962+0000", 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00048260600000000001 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528746+0000", 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027839470000000002 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529350+0000", 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00060369600000000003 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529392+0000", 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2571000000000003e-05 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.014 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 37/31 ECSubWrite(tid=24, reqid=client.4615.0:206, at_version=37'15, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.527047+0000", 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3902801680000003, 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020053649999999998, 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527047+0000", 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527047+0000", 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527048+0000", 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7700000000000004e-07 2024-09-16T03:40:21.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527281+0000", 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023318699999999999 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527292+0000", 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0698e-05 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527300+0000", 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2290000000000003e-06 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527350+0000", 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.956e-05 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527358+0000", 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4230000000000005e-06 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529016+0000", 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001657857 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529052+0000", 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6634000000000002e-05 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 37/31 ECSubWrite(tid=25, reqid=client.4615.0:207, at_version=37'16, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.528218+0000", 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.389109425, 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00086669000000000004, 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528218+0000", 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528218+0000", 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528218+0000", 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528231+0000", 2024-09-16T03:40:21.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2411e-05 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528240+0000", 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9490000000000002e-06 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528245+0000", 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7990000000000001e-06 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528279+0000", 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4023000000000003e-05 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528283+0000", 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5619999999999997e-06 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529057+0000", 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077346700000000001 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529085+0000", 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7939000000000001e-05 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 37/31 ECSubWrite(tid=26, reqid=client.4615.0:207, at_version=0'0, trim_to=0'0, roll_forward_to=37'16))", 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.531427+0000", 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3859004800000001, 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012953559999999999, 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.024 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531427+0000", 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531427+0000", 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531427+0000", 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0500000000000004e-07 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531435+0000", 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7260000000000002e-06 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531442+0000", 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0659999999999998e-06 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531446+0000", 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2529999999999998e-06 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531480+0000", 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3849000000000001e-05 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531487+0000", 2024-09-16T03:40:21.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3509999999999998e-06 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.532692+0000", 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012059799999999999 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.532722+0000", 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9626000000000001e-05 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.028 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s3 37/31 ECSubWrite(tid=14, reqid=client.4615.0:220, at_version=37'10, trim_to=0'0, roll_forward_to=34'8))", 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.571360+0000", 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3459673930000005, 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012286090000000001, 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571360+0000", 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571360+0000", 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571360+0000", 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2100000000000001e-07 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571382+0000", 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1616000000000001e-05 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571390+0000", 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4449999999999996e-06 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571396+0000", 2024-09-16T03:40:21.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7579999999999998e-06 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571451+0000", 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5702000000000003e-05 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571460+0000", 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1189999999999997e-06 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.572555+0000", 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010954579999999999 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.572588+0000", 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3389999999999997e-05 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s3 37/31 ECSubWrite(tid=15, reqid=client.4615.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.573038+0000", 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3442894899999995, 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00082175200000000003, 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573038+0000", 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573038+0000", 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573038+0000", 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4300000000000003e-07 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573048+0000", 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1549999999999996e-06 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573058+0000", 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0597999999999999e-05 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573066+0000", 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4460000000000004e-06 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573121+0000", 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5868000000000001e-05 2024-09-16T03:40:21.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573130+0000", 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9169999999999995e-06 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573815+0000", 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00068421300000000003 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573859+0000", 2024-09-16T03:40:21.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4912e-05 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:225 3.fs0 3:fe9db9b8:::smithi12138290-24:head [append 0~720896] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.538459+0000", 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.3788685699999998, 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076796640000000001, 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:21.038 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 225 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.538459+0000", 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.538459+0000", 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.538462+0000", 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2990000000000001e-06 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539454+0000", 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00099185099999999997 2024-09-16T03:40:21.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539477+0000", 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3056000000000001e-05 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539487+0000", 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0019e-05 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539564+0000", 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6762999999999997e-05 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539570+0000", 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5359999999999998e-06 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541406+0000", 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001836086 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541468+0000", 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1659e-05 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542536+0000", 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001067597 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545013+0000", 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002477061 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.546056+0000", 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00104386 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.546138+0000", 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1877000000000002e-05 2024-09-16T03:40:21.044 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:226 3.fs0 3:fe9db9b8:::smithi12138290-24:head [append 0~376832] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.3768731380000001, 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095498700000000002, 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.045 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 226 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540455+0000", 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540861+0000", 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040549800000000001 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540874+0000", 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2974e-05 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540882+0000", 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540948+0000", 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6383999999999993e-05 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540954+0000", 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9730000000000002e-06 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543210+0000", 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022561209999999998 2024-09-16T03:40:21.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543290+0000", 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9907999999999995e-05 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543922+0000", 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000631713 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545016+0000", 2024-09-16T03:40:21.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001093896 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.549959+0000", 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049431980000000002 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550004+0000", 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4858999999999999e-05 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:227 3.fs0 3:fe9db9b8:::smithi12138290-24:head [setxattr _header (56) in=7b] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.376419496, 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092805639999999998, 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 227 2024-09-16T03:40:21.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540909+0000", 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6400000000000001e-07 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540912+0000", 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1439999999999999e-06 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540919+0000", 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4000000000000003e-06 2024-09-16T03:40:21.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540925+0000", 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6069999999999998e-06 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541006+0000", 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1231999999999997e-05 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:21.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541012+0000", 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3609999999999997e-06 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544371+0000", 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033583739999999999 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544424+0000", 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3535000000000002e-05 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544709+0000", 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028452099999999998 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545291+0000", 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00058281499999999998 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550110+0000", 2024-09-16T03:40:21.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048181430000000004 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550188+0000", 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8668e-05 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:228 3.fs0 3:fe9db9b8:::smithi12138290-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:21.058 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.3763783419999998, 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011210417, 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 228 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0399999999999996e-07 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540952+0000", 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6860000000000002e-06 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540960+0000", 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9729999999999997e-06 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540966+0000", 2024-09-16T03:40:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7509999999999997e-06 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541041+0000", 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5216999999999995e-05 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541046+0000", 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0000000000000004e-06 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544875+0000", 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038287339999999999 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544964+0000", 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9245000000000004e-05 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550540+0000", 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055763649999999998 2024-09-16T03:40:21.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550618+0000", 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7908e-05 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552159+0000", 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015409340000000001 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.fs0 38/31 ECSubRead(tid=66, to_read={3:fe9db9b8:::smithi12138290-24:head=0,4096,0}, subchunks={3:fe9db9b8:::smithi12138290-24:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.550750+0000", 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.3665774690000001, 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00083386400000000002, 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550750+0000", 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550750+0000", 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550750+0000", 2024-09-16T03:40:21.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550750+0000", 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550763+0000", 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3061000000000001e-05 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550776+0000", 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3097e-05 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550849+0000", 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3436000000000006e-05 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.551584+0000", 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073426999999999997 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.068 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 38/31 ECSubWriteReply(tid=65, last_complete=38'11, committed=1, applied=1))", 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.550864+0000", 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.3664635550000002, 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077849699999999996, 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550864+0000", 2024-09-16T03:40:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550864+0000", 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550864+0000", 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.13e-07 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550867+0000", 2024-09-16T03:40:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0680000000000001e-06 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550875+0000", 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3819999999999998e-06 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550879+0000", 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5739999999999997e-06 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.551623+0000", 2024-09-16T03:40:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074388399999999999 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.551642+0000", 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9076000000000002e-05 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.7s6 38/31 ECSubWrite(tid=10, reqid=client.4615.0:231, at_version=38'6, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.552016+0000", 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.3653111459999998, 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013400899999999999, 2024-09-16T03:40:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552016+0000", 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552016+0000", 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552017+0000", 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4899999999999995e-07 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552345+0000", 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000328647 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552360+0000", 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4854e-05 2024-09-16T03:40:21.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552370+0000", 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0246e-05 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552410+0000", 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9464e-05 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552415+0000", 2024-09-16T03:40:21.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.558e-06 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.553323+0000", 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000907101 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.553356+0000", 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3670999999999997e-05 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.076 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.7s6 38/31 ECSubWrite(tid=11, reqid=client.4615.0:232, at_version=38'7, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.553894+0000", 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.3634327900000001, 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00096279500000000001, 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.077 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.553894+0000", 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.553894+0000", 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.553896+0000", 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.28e-06 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554079+0000", 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018375999999999999 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554093+0000", 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3484e-05 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554102+0000", 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7560000000000005e-06 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554173+0000", 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1686000000000004e-05 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554181+0000", 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9910000000000002e-06 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554830+0000", 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000648845 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554857+0000", 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6993000000000001e-05 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.081 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.082 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.082 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.082 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.082 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:21.082 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:40:21.083 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:21.083 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:21.083 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:21.084 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:21.084 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:21.084 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:21.084 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:21.085 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:21.085 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:21.086 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:21.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:21.000+0000 7fdf23fff640 0 -- 172.21.15.76:0/218463064 >> [v2:172.21.15.119:6800/1147733023,v1:172.21.15.119:6801/1147733023] conn(0x7fdf2421e440 msgr2=0x7fdf24220830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:21.087 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:21.085+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:21.187 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:21.186+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:21.191 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:21.191 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:21.191 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:21.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:40:21.221 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:21.221 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:21.238 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:21.238 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:21.288 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:21.287+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:21.307 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:21.307 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:21.307 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:21.307 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:21.307 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:21.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:40:21.381 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:40:21.382 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:40:21.389 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:21.387+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:21.394 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-16T03:40:20.918986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25870899,"num_objects":53,"num_object_clones":4,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":2,"num_read":269,"num_read_kb":8628,"num_write":360,"num_write_kb":30998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":483124,"kb_used_data":50632,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509466316,"statfs":{"total":1546188226560,"available":1545693507584,"internally_reserved":0,"allocated":51847168,"data_stored":49732280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25518,"internal_metadata":442473554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17989632,"num_objects":30,"num_object_clones":4,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":2,"num_read":86,"num_read_kb":7624,"num_write":109,"num_write_kb":19280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000592"},"pg_stats":[{"pgid":"3.f","version":"34'7","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.518710+0000","last_change":"2024-09-16T03:40:15.944882+0000","last_active":"2024-09-16T03:40:16.518710+0000","last_peered":"2024-09-16T03:40:16.518710+0000","last_clean":"2024-09-16T03:40:16.518710+0000","last_became_active":"2024-09-16T03:40:11.518147+0000","last_became_peered":"2024-09-16T03:40:11.518147+0000","last_unstale":"2024-09-16T03:40:16.518710+0000","last_undegraded":"2024-09-16T03:40:16.518710+0000","last_fullsized":"2024-09-16T03:40:16.518710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:31.316358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038085899999999999,"stat_sum":{"num_bytes":1458176,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":2308,"num_write":7,"num_write_kb":1808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,11,7,1,2],"acting":[0,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"37'9","reported_seq":43,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.561165+0000","last_change":"2024-09-16T03:40:15.514473+0000","last_active":"2024-09-16T03:40:16.561165+0000","last_peered":"2024-09-16T03:40:16.561165+0000","last_clean":"2024-09-16T03:40:16.561165+0000","last_became_active":"2024-09-16T03:40:11.516892+0000","last_became_peered":"2024-09-16T03:40:11.516892+0000","last_unstale":"2024-09-16T03:40:16.561165+0000","last_undegraded":"2024-09-16T03:40:16.561165+0000","last_fullsized":"2024-09-16T03:40:16.561165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:34.615043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050673299999999999,"stat_sum":{"num_bytes":1605632,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":7,"num_write_kb":1568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"34'6","reported_seq":42,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.520805+0000","last_change":"2024-09-16T03:40:15.514280+0000","last_active":"2024-09-16T03:40:16.520805+0000","last_peered":"2024-09-16T03:40:16.520805+0000","last_clean":"2024-09-16T03:40:16.520805+0000","last_became_active":"2024-09-16T03:40:11.516335+0000","last_became_peered":"2024-09-16T03:40:11.516335+0000","last_unstale":"2024-09-16T03:40:16.520805+0000","last_undegraded":"2024-09-16T03:40:16.520805+0000","last_fullsized":"2024-09-16T03:40:16.520805+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.881803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048962500000000004,"stat_sum":{"num_bytes":1376256,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":6,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,0,8,2,6],"acting":[12,4,9,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"37'22","reported_seq":70,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.522147+0000","last_change":"2024-09-16T03:40:15.944829+0000","last_active":"2024-09-16T03:40:16.522147+0000","last_peered":"2024-09-16T03:40:16.522147+0000","last_clean":"2024-09-16T03:40:16.522147+0000","last_became_active":"2024-09-16T03:40:11.515959+0000","last_became_peered":"2024-09-16T03:40:11.515959+0000","last_unstale":"2024-09-16T03:40:16.522147+0000","last_undegraded":"2024-09-16T03:40:16.522147+0000","last_fullsized":"2024-09-16T03:40:16.522147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:39.630489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037453299999999997,"stat_sum":{"num_bytes":5324800,"num_objects":8,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":25,"num_read_kb":3659,"num_write":21,"num_write_kb":5200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,1,9,2,6],"acting":[0,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"34'10","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T03:40:15.514056+0000","last_change":"2024-09-16T03:40:15.514439+0000","last_active":"2024-09-16T03:40:15.514056+0000","last_peered":"2024-09-16T03:40:15.514056+0000","last_clean":"2024-09-16T03:40:15.514056+0000","last_became_active":"2024-09-16T03:40:11.502708+0000","last_became_peered":"2024-09-16T03:40:11.502708+0000","last_unstale":"2024-09-16T03:40:15.514056+0000","last_undegraded":"2024-09-16T03:40:15.514056+0000","last_fullsized":"2024-09-16T03:40:15.514056+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.595377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052483800000000002,"stat_sum":{"num_bytes":2392064,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":10,"num_write_kb":2336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"34'5","reported_seq":40,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.520777+0000","last_change":"2024-09-16T03:40:15.514223+0000","last_active":"2024-09-16T03:40:16.520777+0000","last_peered":"2024-09-16T03:40:16.520777+0000","last_clean":"2024-09-16T03:40:16.520777+0000","last_became_active":"2024-09-16T03:40:11.523925+0000","last_became_peered":"2024-09-16T03:40:11.523925+0000","last_unstale":"2024-09-16T03:40:16.520777+0000","last_undegraded":"2024-09-16T03:40:16.520777+0000","last_fullsized":"2024-09-16T03:40:16.520777+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:11.889106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049971599999999998,"stat_sum":{"num_bytes":344064,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"34'9","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.545566+0000","last_change":"2024-09-16T03:40:15.514747+0000","last_active":"2024-09-16T03:40:16.545566+0000","last_peered":"2024-09-16T03:40:16.545566+0000","last_clean":"2024-09-16T03:40:16.545566+0000","last_became_active":"2024-09-16T03:40:11.502278+0000","last_became_peered":"2024-09-16T03:40:11.502278+0000","last_unstale":"2024-09-16T03:40:16.545566+0000","last_undegraded":"2024-09-16T03:40:16.545566+0000","last_fullsized":"2024-09-16T03:40:16.545566+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:24.894273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077272599999999995,"stat_sum":{"num_bytes":196608,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":28,"num_write_kb":864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.8","version":"34'2","reported_seq":36,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.528545+0000","last_change":"2024-09-16T03:40:15.981313+0000","last_active":"2024-09-16T03:40:16.528545+0000","last_peered":"2024-09-16T03:40:16.528545+0000","last_clean":"2024-09-16T03:40:16.528545+0000","last_became_active":"2024-09-16T03:40:11.516188+0000","last_became_peered":"2024-09-16T03:40:11.516188+0000","last_unstale":"2024-09-16T03:40:16.528545+0000","last_undegraded":"2024-09-16T03:40:16.528545+0000","last_fullsized":"2024-09-16T03:40:16.528545+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:38.205204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046374699999999998,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,0,5],"acting":[11,10,1,15,14,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":230,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.528611+0000","last_change":"2024-09-16T03:39:58.261761+0000","last_active":"2024-09-16T03:40:16.528611+0000","last_peered":"2024-09-16T03:40:16.528611+0000","last_clean":"2024-09-16T03:40:16.528611+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:40:16.528611+0000","last_undegraded":"2024-09-16T03:40:16.528611+0000","last_fullsized":"2024-09-16T03:40:16.528611+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.270453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":40,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.528131+0000","last_change":"2024-09-16T03:40:00.965641+0000","last_active":"2024-09-16T03:40:16.528131+0000","last_peered":"2024-09-16T03:40:16.528131+0000","last_clean":"2024-09-16T03:40:16.528131+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:40:16.528131+0000","last_undegraded":"2024-09-16T03:40:16.528131+0000","last_fullsized":"2024-09-16T03:40:16.528131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:22.133279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"34'8","reported_seq":44,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.528676+0000","last_change":"2024-09-16T03:40:15.981435+0000","last_active":"2024-09-16T03:40:16.528676+0000","last_peered":"2024-09-16T03:40:16.528676+0000","last_clean":"2024-09-16T03:40:16.528676+0000","last_became_active":"2024-09-16T03:40:11.524110+0000","last_became_peered":"2024-09-16T03:40:11.524110+0000","last_unstale":"2024-09-16T03:40:16.528676+0000","last_undegraded":"2024-09-16T03:40:16.528676+0000","last_fullsized":"2024-09-16T03:40:16.528676+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:28.250710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055615799999999996,"stat_sum":{"num_bytes":1064960,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":8,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0,4,8,3,7],"acting":[11,5,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":39,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.526565+0000","last_change":"2024-09-16T03:40:00.334268+0000","last_active":"2024-09-16T03:40:16.526565+0000","last_peered":"2024-09-16T03:40:16.526565+0000","last_clean":"2024-09-16T03:40:16.526565+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:40:16.526565+0000","last_undegraded":"2024-09-16T03:40:16.526565+0000","last_fullsized":"2024-09-16T03:40:16.526565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:16.834552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"34'11","reported_seq":48,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.039717+0000","last_change":"2024-09-16T03:40:16.039997+0000","last_active":"2024-09-16T03:40:16.039717+0000","last_peered":"2024-09-16T03:40:16.039717+0000","last_clean":"2024-09-16T03:40:16.039717+0000","last_became_active":"2024-09-16T03:40:11.523175+0000","last_became_peered":"2024-09-16T03:40:11.523175+0000","last_unstale":"2024-09-16T03:40:16.039717+0000","last_undegraded":"2024-09-16T03:40:16.039717+0000","last_fullsized":"2024-09-16T03:40:16.039717+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:22.841800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000439135,"stat_sum":{"num_bytes":1392640,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":11,"num_write_kb":1872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":39,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.527092+0000","last_change":"2024-09-16T03:40:00.965529+0000","last_active":"2024-09-16T03:40:16.527092+0000","last_peered":"2024-09-16T03:40:16.527092+0000","last_clean":"2024-09-16T03:40:16.527092+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:40:16.527092+0000","last_undegraded":"2024-09-16T03:40:16.527092+0000","last_fullsized":"2024-09-16T03:40:16.527092+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.759480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"34'9","reported_seq":36,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T03:40:14.552103+0000","last_change":"2024-09-16T03:40:11.524581+0000","last_active":"2024-09-16T03:40:14.552103+0000","last_peered":"2024-09-16T03:40:14.552103+0000","last_clean":"2024-09-16T03:40:14.552103+0000","last_became_active":"2024-09-16T03:40:11.523741+0000","last_became_peered":"2024-09-16T03:40:11.523741+0000","last_unstale":"2024-09-16T03:40:14.552103+0000","last_undegraded":"2024-09-16T03:40:14.552103+0000","last_fullsized":"2024-09-16T03:40:14.552103+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:22.157241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1130496,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":9,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"27'2","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.526694+0000","last_change":"2024-09-16T03:40:00.334487+0000","last_active":"2024-09-16T03:40:16.526694+0000","last_peered":"2024-09-16T03:40:16.526694+0000","last_clean":"2024-09-16T03:40:16.526694+0000","last_became_active":"2024-09-16T03:39:58.564548+0000","last_became_peered":"2024-09-16T03:39:58.564548+0000","last_unstale":"2024-09-16T03:40:16.526694+0000","last_undegraded":"2024-09-16T03:40:16.526694+0000","last_fullsized":"2024-09-16T03:40:16.526694+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:24.220303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"37'10","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.527308+0000","last_change":"2024-09-16T03:40:16.182363+0000","last_active":"2024-09-16T03:40:16.527308+0000","last_peered":"2024-09-16T03:40:16.527308+0000","last_clean":"2024-09-16T03:40:16.527308+0000","last_became_active":"2024-09-16T03:40:11.515529+0000","last_became_peered":"2024-09-16T03:40:11.515529+0000","last_unstale":"2024-09-16T03:40:16.527308+0000","last_undegraded":"2024-09-16T03:40:16.527308+0000","last_fullsized":"2024-09-16T03:40:16.527308+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.211653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042539599999999998,"stat_sum":{"num_bytes":2375680,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":5,"num_read_kb":3,"num_write":9,"num_write_kb":2320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,0,1,4,12],"acting":[15,8,2,0,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.5","version":"34'9","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.526826+0000","last_change":"2024-09-16T03:40:16.279105+0000","last_active":"2024-09-16T03:40:16.526826+0000","last_peered":"2024-09-16T03:40:16.526826+0000","last_clean":"2024-09-16T03:40:16.526826+0000","last_became_active":"2024-09-16T03:40:11.518878+0000","last_became_peered":"2024-09-16T03:40:11.518878+0000","last_unstale":"2024-09-16T03:40:16.526826+0000","last_undegraded":"2024-09-16T03:40:16.526826+0000","last_fullsized":"2024-09-16T03:40:16.526826+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:29.453812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036837000000000003,"stat_sum":{"num_bytes":1671168,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":9,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7,11,10,2,5],"acting":[8,6,7,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.039457+0000","last_change":"2024-09-16T03:40:01.115542+0000","last_active":"2024-09-16T03:40:16.039457+0000","last_peered":"2024-09-16T03:40:16.039457+0000","last_clean":"2024-09-16T03:40:16.039457+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:40:16.039457+0000","last_undegraded":"2024-09-16T03:40:16.039457+0000","last_fullsized":"2024-09-16T03:40:16.039457+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.045739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"34'4","reported_seq":36,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T03:40:15.514092+0000","last_change":"2024-09-16T03:40:15.514448+0000","last_active":"2024-09-16T03:40:15.514092+0000","last_peered":"2024-09-16T03:40:15.514092+0000","last_clean":"2024-09-16T03:40:15.514092+0000","last_became_active":"2024-09-16T03:40:11.523729+0000","last_became_peered":"2024-09-16T03:40:11.523729+0000","last_unstale":"2024-09-16T03:40:15.514092+0000","last_undegraded":"2024-09-16T03:40:15.514092+0000","last_fullsized":"2024-09-16T03:40:15.514092+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:33.062589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047070599999999999,"stat_sum":{"num_bytes":966656,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.529669+0000","last_change":"2024-09-16T03:40:00.333609+0000","last_active":"2024-09-16T03:40:16.529669+0000","last_peered":"2024-09-16T03:40:16.529669+0000","last_clean":"2024-09-16T03:40:16.529669+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:40:16.529669+0000","last_undegraded":"2024-09-16T03:40:16.529669+0000","last_fullsized":"2024-09-16T03:40:16.529669+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.665433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"34'4","reported_seq":42,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.546880+0000","last_change":"2024-09-16T03:40:16.261046+0000","last_active":"2024-09-16T03:40:16.546880+0000","last_peered":"2024-09-16T03:40:16.546880+0000","last_clean":"2024-09-16T03:40:16.546880+0000","last_became_active":"2024-09-16T03:40:11.518056+0000","last_became_peered":"2024-09-16T03:40:11.518056+0000","last_unstale":"2024-09-16T03:40:16.546880+0000","last_undegraded":"2024-09-16T03:40:16.546880+0000","last_fullsized":"2024-09-16T03:40:16.546880+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:21.179931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031918799999999998,"stat_sum":{"num_bytes":999424,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":1635,"num_write":4,"num_write_kb":976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,13,7,5,11,6,0],"acting":[9,13,7,5,11,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.545583+0000","last_change":"2024-09-16T03:40:10.261076+0000","last_active":"2024-09-16T03:40:16.545583+0000","last_peered":"2024-09-16T03:40:16.545583+0000","last_clean":"2024-09-16T03:40:16.545583+0000","last_became_active":"2024-09-16T03:40:10.260825+0000","last_became_peered":"2024-09-16T03:40:10.260825+0000","last_unstale":"2024-09-16T03:40:16.545583+0000","last_undegraded":"2024-09-16T03:40:16.545583+0000","last_fullsized":"2024-09-16T03:40:16.545583+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:02.522441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3],"acting":[13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.6","version":"37'16","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T03:40:19.536048+0000","last_change":"2024-09-16T03:40:16.231139+0000","last_active":"2024-09-16T03:40:19.536048+0000","last_peered":"2024-09-16T03:40:19.536048+0000","last_clean":"2024-09-16T03:40:19.536048+0000","last_became_active":"2024-09-16T03:40:11.517018+0000","last_became_peered":"2024-09-16T03:40:11.517018+0000","last_unstale":"2024-09-16T03:40:19.536048+0000","last_undegraded":"2024-09-16T03:40:19.536048+0000","last_fullsized":"2024-09-16T03:40:19.536048+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:39.595192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043185000000000001,"stat_sum":{"num_bytes":2441216,"num_objects":7,"num_object_clones":1,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":17,"num_read_kb":872,"num_write":15,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,0,1],"acting":[14,7,15,8,11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":39,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T03:40:16.526703+0000","last_change":"2024-09-16T03:40:00.334076+0000","last_active":"2024-09-16T03:40:16.526703+0000","last_peered":"2024-09-16T03:40:16.526703+0000","last_clean":"2024-09-16T03:40:16.526703+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:40:16.526703+0000","last_undegraded":"2024-09-16T03:40:16.526703+0000","last_fullsized":"2024-09-16T03:40:16.526703+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:14.276919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":25280512,"num_objects":49,"num_object_clones":4,"num_object_copies":343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":2,"num_read":133,"num_read_kb":8510,"num_write":155,"num_write_kb":27536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45047808,"data_stored":47763456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247816,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29708,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342132,"statfs":{"total":96636764160,"available":96606343168,"internally_reserved":0,"allocated":2744320,"data_stored":2522117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247815,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28808,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343032,"statfs":{"total":96636764160,"available":96607264768,"internally_reserved":0,"allocated":1814528,"data_stored":1592325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":23,"seq":98784247816,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29756,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342084,"statfs":{"total":96636764160,"available":96606294016,"internally_reserved":0,"allocated":2793472,"data_stored":2565126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313224,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29356,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342484,"statfs":{"total":96636764160,"available":96606703616,"internally_reserved":0,"allocated":2371584,"data_stored":2425864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,5,7,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280519,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31656,"kb_used_data":4628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340184,"statfs":{"total":96636764160,"available":96604348416,"internally_reserved":0,"allocated":4739072,"data_stored":4522939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280519,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30964,"kb_used_data":3932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340876,"statfs":{"total":96636764160,"available":96605057024,"internally_reserved":0,"allocated":4026368,"data_stored":3810235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30412,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341428,"statfs":{"total":96636764160,"available":96605622272,"internally_reserved":0,"allocated":3465216,"data_stored":3245006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[1,3,4,5,6,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247815,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28620,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343220,"statfs":{"total":96636764160,"available":96607457280,"internally_reserved":0,"allocated":1630208,"data_stored":1414075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31260,"kb_used_data":4232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340580,"statfs":{"total":96636764160,"available":96604753920,"internally_reserved":0,"allocated":4333568,"data_stored":4199430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247815,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30816,"kb_used_data":3784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341024,"statfs":{"total":96636764160,"available":96605208576,"internally_reserved":0,"allocated":3874816,"data_stored":3652613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30616,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341224,"statfs":{"total":96636764160,"available":96605413376,"internally_reserved":0,"allocated":3674112,"data_stored":3838982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247814,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30008,"kb_used_data":2976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341832,"statfs":{"total":96636764160,"available":96606035968,"internally_reserved":0,"allocated":3047424,"data_stored":3008038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29460,"kb_used_data":2432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342380,"statfs":{"total":96636764160,"available":96606597120,"internally_reserved":0,"allocated":2490368,"data_stored":2274235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29440,"kb_used_data":2412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342400,"statfs":{"total":96636764160,"available":96606617600,"internally_reserved":0,"allocated":2469888,"data_stored":2249678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,7,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30368,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341472,"statfs":{"total":96636764160,"available":96605667328,"internally_reserved":0,"allocated":3411968,"data_stored":3195835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,4,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247815,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":4844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":4960256,"data_stored":5215782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3895296,"data_stored":3989504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4333568,"data_stored":4333568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3452928,"data_stored":3645440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1224704,"data_stored":1224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3235840,"data_stored":3629056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2076672,"data_stored":2269184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3620864,"data_stored":3620864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2342912,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2318336,"data_stored":2412544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3006464,"data_stored":3198976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3055616,"data_stored":3055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3989504,"data_stored":4476928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2355200,"data_stored":2355200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1392640,"data_stored":1585152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1900544,"data_stored":2195456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":2846720,"data_stored":3334144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:21.396 INFO:tasks.thrashosds.thrasher:Setting 2.2 to [14, 3] 2024-09-16T03:40:21.396 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.2', '14', '3'] 2024-09-16T03:40:21.396 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.2 14 3 2024-09-16T03:40:21.423 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:21.423 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:21.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:21.423 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:21.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.423 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 37/31 ECSubWrite(tid=26, reqid=client.4615.0:213, at_version=37'12, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:21.423 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.554718+0000", 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8663251980000002, 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014147280000000001, 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.554718+0000", 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.554718+0000", 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.554718+0000", 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.46e-07 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555147+0000", 2024-09-16T03:40:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00042841 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555157+0000", 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9189999999999994e-06 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555180+0000", 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3561999999999999e-05 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555225+0000", 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5281999999999998e-05 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555233+0000", 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.576e-06 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556107+0000", 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00087447499999999999 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556133+0000", 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5157999999999999e-05 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=25, reqid=client.4615.0:217, at_version=37'7, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.555382+0000", 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8656606729999998, 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010018010000000001, 2024-09-16T03:40:21.428 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555382+0000", 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555382+0000", 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555383+0000", 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.02e-07 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555397+0000", 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4606999999999999e-05 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555406+0000", 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5080000000000005e-06 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555413+0000", 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.289e-06 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555477+0000", 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3931999999999997e-05 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555486+0000", 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5179999999999993e-06 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556355+0000", 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00086894499999999998 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556384+0000", 2024-09-16T03:40:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8900000000000001e-05 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=26, reqid=client.4615.0:218, at_version=37'8, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.556488+0000", 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8645547259999997, 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056059000000000002, 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556488+0000", 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556488+0000", 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556489+0000", 2024-09-16T03:40:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556500+0000", 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1535999999999999e-05 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556507+0000", 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6050000000000003e-06 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556512+0000", 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1449999999999999e-06 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556559+0000", 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6561999999999999e-05 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556566+0000", 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.199e-06 2024-09-16T03:40:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557026+0000", 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00046014199999999997 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557049+0000", 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3099999999999999e-05 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 37/31 ECSubWrite(tid=27, reqid=client.4615.0:214, at_version=37'13, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.556843+0000", 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8641996389999997, 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00078009200000000002, 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556843+0000", 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556843+0000", 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556844+0000", 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556982+0000", 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138009 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556991+0000", 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.867e-06 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556996+0000", 2024-09-16T03:40:21.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.361e-06 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557033+0000", 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6655000000000001e-05 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557038+0000", 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8029999999999996e-06 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557601+0000", 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056317100000000001 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557624+0000", 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2665999999999999e-05 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=27, reqid=client.4615.0:219, at_version=37'9, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:21.444 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.557360+0000", 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8636832390000002, 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001050537, 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557360+0000", 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557360+0000", 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557360+0000", 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2799999999999996e-07 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557372+0000", 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1201e-05 2024-09-16T03:40:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557380+0000", 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0369999999999995e-06 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557386+0000", 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1380000000000004e-06 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557422+0000", 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6093000000000002e-05 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557428+0000", 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9939999999999997e-06 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.558381+0000", 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00095336599999999998 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.558410+0000", 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9179999999999998e-05 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 37/31 ECSubWrite(tid=28, reqid=client.4615.0:215, at_version=37'14, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.557971+0000", 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8630722080000002, 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013109319999999999, 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557971+0000", 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557971+0000", 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557971+0000", 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3799999999999997e-07 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557986+0000", 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4572e-05 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557994+0000", 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.824e-06 2024-09-16T03:40:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557998+0000", 2024-09-16T03:40:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5260000000000004e-06 2024-09-16T03:40:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.558036+0000", 2024-09-16T03:40:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7755e-05 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.558041+0000", 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2440000000000001e-06 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.559256+0000", 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012145929999999999 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.559282+0000", 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.588e-05 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 37/31 ECSubWrite(tid=29, reqid=client.4615.0:215, at_version=0'0, trim_to=0'0, roll_forward_to=37'14))", 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.561438+0000", 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.859605234, 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043949800000000003, 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561438+0000", 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561438+0000", 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561438+0000", 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561448+0000", 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561458+0000", 2024-09-16T03:40:21.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0086e-05 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561464+0000", 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4489999999999998e-06 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561509+0000", 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5121999999999999e-05 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561514+0000", 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3770000000000004e-06 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561846+0000", 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033198400000000001 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561877+0000", 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1195000000000002e-05 2024-09-16T03:40:21.458 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=28, reqid=client.4615.0:219, at_version=0'0, trim_to=0'0, roll_forward_to=37'9))", 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.561461+0000", 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8595819530000002, 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012191909999999999, 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561461+0000", 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561461+0000", 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561462+0000", 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561469+0000", 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6429999999999995e-06 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561479+0000", 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6770000000000007e-06 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561487+0000", 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4270000000000002e-06 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561538+0000", 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1001000000000002e-05 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561543+0000", 2024-09-16T03:40:21.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7019999999999998e-06 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.562657+0000", 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011138770000000001 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.562680+0000", 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3235e-05 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 37/31 ECSubWrite(tid=14, reqid=client.4615.0:220, at_version=37'10, trim_to=0'0, roll_forward_to=34'8))", 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.571440+0000", 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8496030719999998, 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001255831, 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571440+0000", 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571440+0000", 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571440+0000", 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2499999999999999e-07 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571461+0000", 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.065e-05 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571469+0000", 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2560000000000002e-06 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571475+0000", 2024-09-16T03:40:21.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6729999999999997e-06 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571533+0000", 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8482e-05 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571542+0000", 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2820000000000002e-06 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.572664+0000", 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001121562 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.572696+0000", 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1801000000000003e-05 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 37/31 ECSubWrite(tid=15, reqid=client.4615.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.573055+0000", 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8479878110000003, 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00069386000000000003, 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573055+0000", 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573055+0000", 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573056+0000", 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7299999999999995e-07 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573066+0000", 2024-09-16T03:40:21.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0499999999999999e-05 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573081+0000", 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4219999999999999e-05 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573089+0000", 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.225e-06 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573144+0000", 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5498999999999998e-05 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573154+0000", 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0087e-05 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573719+0000", 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056409400000000001 2024-09-16T03:40:21.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573749+0000", 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0562000000000003e-05 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.0s2 37/31 ECSubRead(tid=17, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.573727+0000", 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8473164889999998, 2024-09-16T03:40:21.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00046106499999999998, 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573727+0000", 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573727+0000", 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573727+0000", 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573735+0000", 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7810000000000005e-06 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573745+0000", 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0229e-05 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573767+0000", 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2050999999999999e-05 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573834+0000", 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7219999999999997e-05 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.574188+0000", 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00035342799999999998 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.0s2 37/31 ECSubRead(tid=18, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.573890+0000", 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8471533759999996, 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000398673, 2024-09-16T03:40:21.477 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573890+0000", 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573890+0000", 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573890+0000", 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3200000000000005e-07 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573897+0000", 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.165e-06 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573914+0000", 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6591000000000001e-05 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573924+0000", 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0363000000000001e-05 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.574229+0000", 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030459299999999998 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.574288+0000", 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9429000000000003e-05 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.0s2 37/31 ECSubRead(tid=19, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.577274+0000", 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8437689449999999, 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016012599999999999, 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.481 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.577274+0000", 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.577274+0000", 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.577275+0000", 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6700000000000003e-07 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.577281+0000", 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5549999999999999e-06 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.577293+0000", 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1955e-05 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.577300+0000", 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919999999999998e-06 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.577374+0000", 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4647999999999994e-05 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.577434+0000", 2024-09-16T03:40:21.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9809000000000003e-05 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.es3 38/31 ECSubRead(tid=29, to_read={3:7acde04a:::smithi12138290-46:head=0,192512,0}, subchunks={3:7acde04a:::smithi12138290-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.545569+0000", 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.8754739799999998, 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000662844, 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545569+0000", 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545569+0000", 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545569+0000", 2024-09-16T03:40:21.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5100000000000001e-07 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545579+0000", 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3600000000000002e-06 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545594+0000", 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5387e-05 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545604+0000", 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0182e-05 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545683+0000", 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9118999999999999e-05 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.546232+0000", 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000548545 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.488 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.489 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.489 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.489 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.489 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:21.490 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:40:21.491 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:21.488+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:21.557 INFO:tasks.rados.rados.0.smithi121.stdout:97: done (4 left) 2024-09-16T03:40:21.557 INFO:tasks.rados.rados.0.smithi121.stdout:98: copy_from oid 42 from oid 8 current snap is 5 2024-09-16T03:40:21.557 INFO:tasks.rados.rados.0.smithi121.stdout:92: finishing write tid 1 to smithi12138290-33 2024-09-16T03:40:21.557 INFO:tasks.rados.rados.0.smithi121.stdout:93: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-16T03:40:21.580 INFO:tasks.rados.rados.0.smithi121.stdout:92: finishing write tid 2 to smithi12138290-33 2024-09-16T03:40:21.580 INFO:tasks.rados.rados.0.smithi121.stdout:92: finishing write tid 3 to smithi12138290-33 2024-09-16T03:40:21.580 INFO:tasks.rados.rados.0.smithi121.stdout:93: done (4 left) 2024-09-16T03:40:21.580 INFO:tasks.rados.rados.0.smithi121.stdout:99: read oid 29 snap -1 2024-09-16T03:40:21.580 INFO:tasks.rados.rados.0.smithi121.stdout:99: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T03:40:21.580 INFO:tasks.rados.rados.0.smithi121.stdout:100: delete oid 26 current snap is 5 2024-09-16T03:40:21.580 INFO:tasks.rados.rados.0.smithi121.stdout:92: finishing write tid 4 to smithi12138290-33 2024-09-16T03:40:21.580 INFO:tasks.rados.rados.0.smithi121.stdout:92: oid 33 updating version 0 to 5 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:92: oid 33 updating version 5 to 6 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:92: oid 33 version 6 is already newer than 4 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 33 v 6 (ObjNum 67 snap 5 seq_num 67) dirty exists 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:92: left oid 33 (ObjNum 67 snap 5 seq_num 67) 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:95: finishing copy_from racing read to smithi12138290-43 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:95: finishing copy_from to smithi12138290-43 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 43 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:96: finishing copy_from racing read to smithi12138290-11 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:96: finishing copy_from to smithi12138290-11 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 11 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:98: finishing copy_from racing read to smithi12138290-42 2024-09-16T03:40:21.581 INFO:tasks.rados.rados.0.smithi121.stdout:98: finishing copy_from to smithi12138290-42 2024-09-16T03:40:21.582 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 42 v 4 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:40:21.583 INFO:tasks.rados.rados.0.smithi121.stdout:92: done (5 left) 2024-09-16T03:40:21.583 INFO:tasks.rados.rados.0.smithi121.stdout:95: done (4 left) 2024-09-16T03:40:21.583 INFO:tasks.rados.rados.0.smithi121.stdout:96: done (3 left) 2024-09-16T03:40:21.583 INFO:tasks.rados.rados.0.smithi121.stdout:98: done (2 left) 2024-09-16T03:40:21.583 INFO:tasks.rados.rados.0.smithi121.stdout:100: done (1 left) 2024-09-16T03:40:21.583 INFO:tasks.rados.rados.0.smithi121.stdout:101: read oid 40 snap -1 2024-09-16T03:40:21.583 INFO:tasks.rados.rados.0.smithi121.stdout:101: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-16T03:40:21.583 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 12 current snap is 5 2024-09-16T03:40:21.583 INFO:tasks.rados.rados.0.smithi121.stdout:102: seq_num 70 ranges {196608=802816,999424=376832} 2024-09-16T03:40:21.588 INFO:tasks.rados.rados.0.smithi121.stdout:102: writing smithi12138290-12 from 196608 to 999424 tid 1 2024-09-16T03:40:21.591 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:21.589+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:21.591 INFO:tasks.rados.rados.0.smithi121.stdout:102: writing smithi12138290-12 from 999424 to 1376256 tid 2 2024-09-16T03:40:21.591 INFO:tasks.rados.rados.0.smithi121.stdout:103: copy_from oid 27 from oid 8 current snap is 5 2024-09-16T03:40:21.591 INFO:tasks.rados.rados.0.smithi121.stdout:101: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-16T03:40:21.609 INFO:tasks.rados.rados.0.smithi121.stdout:99: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T03:40:21.621 INFO:tasks.rados.rados.0.smithi121.stdout:102: finishing write tid 1 to smithi12138290-12 2024-09-16T03:40:21.621 INFO:tasks.rados.rados.0.smithi121.stdout:102: finishing write tid 2 to smithi12138290-12 2024-09-16T03:40:21.621 INFO:tasks.rados.rados.0.smithi121.stdout:102: finishing write tid 3 to smithi12138290-12 2024-09-16T03:40:21.621 INFO:tasks.rados.rados.0.smithi121.stdout:102: finishing write tid 4 to smithi12138290-12 2024-09-16T03:40:21.621 INFO:tasks.rados.rados.0.smithi121.stdout:102: oid 12 updating version 0 to 13 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:102: oid 12 version 13 is already newer than 12 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:102: oid 12 updating version 13 to 14 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 14 (ObjNum 70 snap 5 seq_num 70) dirty exists 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:102: left oid 12 (ObjNum 70 snap 5 seq_num 70) 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:103: finishing copy_from racing read to smithi12138290-27 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:99: done (3 left) 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:101: done (2 left) 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:102: done (1 left) 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 1 current snap is 5 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:103: finishing copy_from to smithi12138290-27 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 27 v 4 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:40:21.622 INFO:tasks.rados.rados.0.smithi121.stdout:104: seq_num 71 ranges {32768=425984,458752=573440,1032192=196608} 2024-09-16T03:40:21.623 INFO:tasks.rados.rados.0.smithi121.stdout:104: writing smithi12138290-1 from 32768 to 458752 tid 1 2024-09-16T03:40:21.627 INFO:tasks.rados.rados.0.smithi121.stdout:104: writing smithi12138290-1 from 458752 to 1032192 tid 2 2024-09-16T03:40:21.628 INFO:tasks.rados.rados.0.smithi121.stdout:104: writing smithi12138290-1 from 1032192 to 1228800 tid 3 2024-09-16T03:40:21.628 INFO:tasks.rados.rados.0.smithi121.stdout:103: done (1 left) 2024-09-16T03:40:21.628 INFO:tasks.rados.rados.0.smithi121.stdout:105: read oid 3 snap -1 2024-09-16T03:40:21.628 INFO:tasks.rados.rados.0.smithi121.stdout:105: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-16T03:40:21.628 INFO:tasks.rados.rados.0.smithi121.stdout:106: snap_create 2024-09-16T03:40:21.629 INFO:tasks.rados.rados.0.smithi121.stdout:104: finishing write tid 1 to smithi12138290-1 2024-09-16T03:40:21.632 INFO:tasks.rados.rados.0.smithi121.stdout:105: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-16T03:40:21.634 INFO:tasks.rados.rados.0.smithi121.stdout:104: finishing write tid 2 to smithi12138290-1 2024-09-16T03:40:21.635 INFO:tasks.rados.rados.0.smithi121.stdout:104: finishing write tid 3 to smithi12138290-1 2024-09-16T03:40:21.636 INFO:tasks.rados.rados.0.smithi121.stdout:104: finishing write tid 4 to smithi12138290-1 2024-09-16T03:40:21.637 INFO:tasks.rados.rados.0.smithi121.stdout:104: finishing write tid 5 to smithi12138290-1 2024-09-16T03:40:21.637 INFO:tasks.rados.rados.0.smithi121.stdout:104: oid 1 updating version 0 to 15 2024-09-16T03:40:21.637 INFO:tasks.rados.rados.0.smithi121.stdout:104: oid 1 version 15 is already newer than 14 2024-09-16T03:40:21.637 INFO:tasks.rados.rados.0.smithi121.stdout:104: oid 1 version 15 is already newer than 13 2024-09-16T03:40:21.637 INFO:tasks.rados.rados.0.smithi121.stdout:104: oid 1 updating version 15 to 16 2024-09-16T03:40:21.637 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 1 v 16 (ObjNum 71 snap 5 seq_num 71) dirty exists 2024-09-16T03:40:21.637 INFO:tasks.rados.rados.0.smithi121.stdout:104: left oid 1 (ObjNum 71 snap 5 seq_num 71) 2024-09-16T03:40:21.645 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:21.645 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:21.645 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:21.645 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:21.654 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:40:21.691 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:21.690+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:21.756 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:21.757 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:21.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:40:21.809 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:21.807+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:21.870 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4341.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:39:58.856812+0000", 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.010897850999999, 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022137099999999999, 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4341", 2024-09-16T03:40:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:58.856812+0000", 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:58.856812+0000", 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:58.856815+0000", 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3999999999999999e-06 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:58.856821+0000", 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5039999999999999e-06 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:58.856833+0000", 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1558e-05 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:58.856842+0000", 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2299999999999997e-06 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:58.856912+0000", 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9756000000000001e-05 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:58.857034+0000", 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121923 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4341.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:39:59.266198+0000", 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.601512494000001, 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035726070000000002, 2024-09-16T03:40:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4341", 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266198+0000", 2024-09-16T03:40:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266198+0000", 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266200+0000", 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5620000000000002e-06 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266214+0000", 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3709999999999999e-05 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266236+0000", 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2407000000000001e-05 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266247+0000", 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0152999999999999e-05 2024-09-16T03:40:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266319+0000", 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2133999999999997e-05 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266328+0000", 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0799999999999995e-06 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.268192+0000", 2024-09-16T03:40:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018645529999999999 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.268308+0000", 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011581100000000001 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 10", 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.268416+0000", 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010828300000000001 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269196+0000", 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077989699999999999 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269705+0000", 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00050892999999999999 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269742+0000", 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7128000000000001e-05 2024-09-16T03:40:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269770+0000", 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7959000000000001e-05 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4341.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:00.332280+0000", 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.535430075000001, 2024-09-16T03:40:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034863479999999998, 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4341", 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332280+0000", 2024-09-16T03:40:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332280+0000", 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332286+0000", 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9730000000000002e-06 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332296+0000", 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0019e-05 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332314+0000", 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7331000000000001e-05 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332323+0000", 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.414e-06 2024-09-16T03:40:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332406+0000", 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3013e-05 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332411+0000", 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5480000000000003e-06 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.333895+0000", 2024-09-16T03:40:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014844089999999999 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.333953+0000", 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8242999999999998e-05 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 10", 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.334047+0000", 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3635999999999996e-05 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.334642+0000", 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00059534900000000001 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.335694+0000", 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001051616 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.335728+0000", 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3912999999999999e-05 2024-09-16T03:40:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.335767+0000", 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8884000000000002e-05 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/31 ECSubWrite(tid=23, reqid=client.4615.0:205, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.3425202040000004, 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035619599999999999, 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.31e-07 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525645+0000", 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045454399999999998 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525658+0000", 2024-09-16T03:40:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3159999999999999e-05 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525669+0000", 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1134999999999999e-05 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525736+0000", 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7175000000000007e-05 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525746+0000", 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.927e-06 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528720+0000", 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029734119999999999 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528752+0000", 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2475999999999999e-05 2024-09-16T03:40:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/31 ECSubWrite(tid=24, reqid=client.4615.0:206, at_version=37'15, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.527260+0000", 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.3404501040000003, 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001806607, 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527260+0000", 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527260+0000", 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527261+0000", 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.68e-07 2024-09-16T03:40:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527547+0000", 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000286533 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527562+0000", 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.456e-05 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527571+0000", 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7299999999999994e-06 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527622+0000", 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0868000000000002e-05 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527632+0000", 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0365e-05 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529039+0000", 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001407478 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529067+0000", 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7404999999999999e-05 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/31 ECSubWrite(tid=25, reqid=client.4615.0:207, at_version=37'16, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.528306+0000", 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.3394045710000002, 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00079617999999999996, 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528306+0000", 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528306+0000", 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528306+0000", 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5499999999999998e-07 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528318+0000", 2024-09-16T03:40:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2041e-05 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528328+0000", 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.346e-06 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528333+0000", 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.135e-06 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528380+0000", 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7524000000000001e-05 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528390+0000", 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8279999999999998e-06 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529073+0000", 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00068300999999999995 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529102+0000", 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8741000000000001e-05 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/31 ECSubWrite(tid=26, reqid=client.4615.0:207, at_version=0'0, trim_to=0'0, roll_forward_to=37'16))", 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.531465+0000", 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.3362448819999999, 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00121079, 2024-09-16T03:40:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531465+0000", 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531465+0000", 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531466+0000", 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9299999999999998e-07 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531475+0000", 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3379999999999994e-06 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531485+0000", 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531491+0000", 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531540+0000", 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8968000000000003e-05 2024-09-16T03:40:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.531545+0000", 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.451e-06 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.532655+0000", 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001110062 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.532676+0000", 2024-09-16T03:40:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1154000000000001e-05 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s1 37/31 ECSubWrite(tid=14, reqid=client.4615.0:220, at_version=37'10, trim_to=0'0, roll_forward_to=34'8))", 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.571513+0000", 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.2961970410000001, 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010708029999999999, 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571513+0000", 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571513+0000", 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571514+0000", 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571549+0000", 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5228000000000002e-05 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571560+0000", 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0963e-05 2024-09-16T03:40:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571570+0000", 2024-09-16T03:40:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9019999999999994e-06 2024-09-16T03:40:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571642+0000", 2024-09-16T03:40:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2863999999999999e-05 2024-09-16T03:40:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571652+0000", 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.572552+0000", 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00090006099999999996 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.572584+0000", 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2079999999999998e-05 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s1 37/31 ECSubWrite(tid=15, reqid=client.4615.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.573056+0000", 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.2946543449999997, 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00090859600000000001, 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573056+0000", 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573056+0000", 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573057+0000", 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1099999999999995e-07 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573066+0000", 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8980000000000008e-06 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573081+0000", 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4224e-05 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573092+0000", 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1815e-05 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573144+0000", 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1913999999999998e-05 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573153+0000", 2024-09-16T03:40:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0860000000000008e-06 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573931+0000", 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077769899999999997 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573965+0000", 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3349000000000002e-05 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:229 3.5s0 3:a4bec39a:::smithi12138290-4:head [copy-from ver 9] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3271340450000002, 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016562443, 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 229 2024-09-16T03:40:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540579+0000", 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6910000000000002e-06 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540591+0000", 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2018000000000001e-05 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540631+0000", 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9938999999999998e-05 2024-09-16T03:40:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540643+0000", 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1902000000000001e-05 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540731+0000", 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8585000000000007e-05 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540736+0000", 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9919999999999998e-06 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542533+0000", 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017970950000000001 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542727+0000", 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019304700000000001 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.553519+0000", 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010792568000000001 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554988+0000", 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014687490000000001 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557059+0000", 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00207092 2024-09-16T03:40:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557139+0000", 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9937000000000004e-05 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:230 3.5s0 3:a4bec39a:::smithi12138290-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.32702294, 2024-09-16T03:40:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014084419000000001, 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 230 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:40:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540688+0000", 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540692+0000", 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.019e-06 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540724+0000", 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2237000000000003e-05 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540740+0000", 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6075000000000001e-05 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540783+0000", 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2249000000000002e-05 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540786+0000", 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4869999999999998e-06 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543015+0000", 2024-09-16T03:40:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022290859999999999 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543063+0000", 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7496000000000001e-05 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554628+0000", 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011565203 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554697+0000", 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9381999999999994e-05 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554772+0000", 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4535000000000004e-05 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 38/31 ECSubWriteReply(tid=20, last_complete=38'11, committed=1, applied=1))", 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.556928+0000", 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3107819569999997, 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00052021999999999995, 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.556928+0000", 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.556928+0000", 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.556929+0000", 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9299999999999997e-07 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.556933+0000", 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9519999999999996e-06 2024-09-16T03:40:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.556941+0000", 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7870000000000002e-06 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.556945+0000", 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.403e-06 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.556986+0000", 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0878999999999999e-05 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557449+0000", 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000462406 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/31 ECSubWriteReply(tid=22, last_complete=42'13, committed=1, applied=1))", 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.554532+0000", 2024-09-16T03:40:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.31317834100000003, 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000552477, 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.554532+0000", 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.554532+0000", 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.554533+0000", 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.554537+0000", 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8959999999999996e-06 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.554547+0000", 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0414000000000001e-05 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.554554+0000", 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3089999999999998e-06 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.554645+0000", 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0768999999999998e-05 2024-09-16T03:40:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.555084+0000", 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043941199999999997 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 42/31 ECSubWrite(tid=30, reqid=osd.14.0:30, at_version=42'20, trim_to=0'0, roll_forward_to=41'18))", 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.31013564100000002, 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048714179999999998, 2024-09-16T03:40:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3799999999999999e-07 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557594+0000", 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9415e-05 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557606+0000", 2024-09-16T03:40:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1354e-05 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557615+0000", 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9600000000000006e-06 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557669+0000", 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4024000000000003e-05 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557674+0000", 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.587e-06 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.562413+0000", 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004739254 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.562446+0000", 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2586000000000003e-05 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.6s3 42/31 ECSubRead(tid=31, to_read={3:616ac6f8:::smithi12138290-8:head=0,32768,0}, subchunks={3:616ac6f8:::smithi12138290-8:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.559275+0000", 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.30843512299999998, 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00311509, 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.559275+0000", 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.559275+0000", 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.559276+0000", 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.92e-07 2024-09-16T03:40:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.559285+0000", 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3230000000000004e-06 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.559302+0000", 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6518e-05 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.559312+0000", 2024-09-16T03:40:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0329999999999999e-05 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.559361+0000", 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8965999999999998e-05 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.562390+0000", 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030292610000000001 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 42/31 ECSubWrite(tid=32, reqid=osd.14.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.563702+0000", 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.30400797899999998, 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00044142699999999999, 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.563702+0000", 2024-09-16T03:40:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.563702+0000", 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.563703+0000", 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5700000000000002e-07 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.563711+0000", 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7519999999999996e-06 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.563720+0000", 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.563726+0000", 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4349999999999996e-06 2024-09-16T03:40:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.563773+0000", 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7162e-05 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.563781+0000", 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3580000000000003e-06 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.564119+0000", 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033755299999999999 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.564144+0000", 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4827000000000001e-05 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.3s1 42/31 ECSubRead(tid=16, to_read={3:c98c73f3:::smithi12138290-40:head=0,167936,0}, subchunks={3:c98c73f3:::smithi12138290-40:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.582830+0000", 2024-09-16T03:40:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.28488070799999998, 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024031740000000001, 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582830+0000", 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582830+0000", 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582830+0000", 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.05e-07 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582838+0000", 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8890000000000002e-06 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582853+0000", 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5420000000000001e-05 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582864+0000", 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0611000000000001e-05 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582945+0000", 2024-09-16T03:40:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1538000000000004e-05 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.585233+0000", 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022875109999999999 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.3s1 42/31 ECSubRead(tid=17, to_read={3:c98c73f3:::smithi12138290-40:head=0,167936,0}, subchunks={3:c98c73f3:::smithi12138290-40:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.583748+0000", 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.28396225800000002, 2024-09-16T03:40:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016934470000000001, 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583748+0000", 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583748+0000", 2024-09-16T03:40:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583749+0000", 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7599999999999997e-07 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583757+0000", 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6519999999999995e-06 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583773+0000", 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5554999999999999e-05 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583784+0000", 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1229999999999999e-05 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.585319+0000", 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015344460000000001 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.585441+0000", 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012288800000000001 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.3s1 42/31 ECSubRead(tid=18, to_read={3:c98c73f3:::smithi12138290-40:head=0,167936,0}, subchunks={3:c98c73f3:::smithi12138290-40:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.584054+0000", 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.28365654000000001, 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001503377, 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.584054+0000", 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.584054+0000", 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.584054+0000", 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6700000000000003e-07 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.584062+0000", 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.584076+0000", 2024-09-16T03:40:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4579999999999999e-05 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.584087+0000", 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1214e-05 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.585469+0000", 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013814490000000001 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.585557+0000", 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8256000000000003e-05 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.6s3 42/31 ECSubRead(tid=33, to_read={3:616ac6f8:::smithi12138290-8:head=0,32768,0}, subchunks={3:616ac6f8:::smithi12138290-8:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.597767+0000", 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.26994339899999997, 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00020485600000000001, 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.597767+0000", 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.597767+0000", 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.597768+0000", 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1769999999999999e-06 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.597781+0000", 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2599000000000001e-05 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.597801+0000", 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0749000000000001e-05 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.597813+0000", 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1729000000000001e-05 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.597902+0000", 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8454000000000004e-05 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.597972+0000", 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0148000000000007e-05 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.966 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:21.967 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:21.967 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:21.968 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:40:21.969 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:21.966+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:22.069 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:22.066+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:22.113 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:22.113 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:22.113 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:22.113 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:22.122 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:40:22.169 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:22.168+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:22.267 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:22.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:22.268 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:22.268 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:22.268 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:22.270 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:22.269+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:22.277 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:40:22.372 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:22.370+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:22.381 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:22.381 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:22.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:22.381 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:22.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.381 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/31 ECSubWrite(tid=60, reqid=client.4615.0:209, at_version=37'22, trim_to=0'0, roll_forward_to=34'20))", 2024-09-16T03:40:22.381 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.526075+0000", 2024-09-16T03:40:22.381 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.8526350750000002, 2024-09-16T03:40:22.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002713586, 2024-09-16T03:40:22.381 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526075+0000", 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526075+0000", 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526076+0000", 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5600000000000004e-07 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526103+0000", 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7205000000000001e-05 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526119+0000", 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5874000000000001e-05 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526128+0000", 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5599999999999999e-06 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526187+0000", 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8427000000000002e-05 2024-09-16T03:40:22.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526197+0000", 2024-09-16T03:40:22.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.91e-06 2024-09-16T03:40:22.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528753+0000", 2024-09-16T03:40:22.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025567099999999998 2024-09-16T03:40:22.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528788+0000", 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4944000000000003e-05 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=14, reqid=client.4615.0:220, at_version=37'10, trim_to=0'0, roll_forward_to=34'8))", 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.571551+0000", 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.8071589619999999, 2024-09-16T03:40:22.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001182958, 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571551+0000", 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571551+0000", 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571551+0000", 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4499999999999998e-07 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571573+0000", 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1868000000000001e-05 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571582+0000", 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5480000000000002e-06 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571590+0000", 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8839999999999994e-06 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571660+0000", 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0664e-05 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571674+0000", 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.34e-05 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.572695+0000", 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001021559 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.572734+0000", 2024-09-16T03:40:22.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.879e-05 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/31 ECSubWrite(tid=15, reqid=client.4615.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.573084+0000", 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.8056256749999999, 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077967800000000001, 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573084+0000", 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573084+0000", 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573085+0000", 2024-09-16T03:40:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573097+0000", 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2054e-05 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573111+0000", 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4503e-05 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573120+0000", 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1259999999999998e-06 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573190+0000", 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0648999999999994e-05 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573199+0000", 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.935e-06 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573834+0000", 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00063481500000000005 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573864+0000", 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9932000000000001e-05 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:253 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79965165400000005, 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084206999999999997, 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 253 2024-09-16T03:40:22.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579061+0000", 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3350000000000001e-06 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579072+0000", 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0964999999999999e-05 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579093+0000", 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1766000000000001e-05 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579104+0000", 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0271999999999999e-05 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579200+0000", 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5917999999999999e-05 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579208+0000", 2024-09-16T03:40:22.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9479999999999997e-06 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581270+0000", 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020611140000000002 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581437+0000", 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016734000000000001 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587479+0000", 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060420420000000001 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:254 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79956856799999998, 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087869469999999998, 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.401 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 254 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579142+0000", 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579146+0000", 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0600000000000001e-06 2024-09-16T03:40:22.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579155+0000", 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7620000000000001e-06 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579161+0000", 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8640000000000001e-06 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579289+0000", 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000128677 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579298+0000", 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4100000000000008e-06 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581916+0000", 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026177570000000001 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581995+0000", 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9833999999999995e-05 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587928+0000", 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059329329999999996 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.406 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:255 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79952465100000003, 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090005769999999992, 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 255 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.407 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579186+0000", 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9500000000000002e-07 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579189+0000", 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579197+0000", 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8189999999999992e-06 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579202+0000", 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579337+0000", 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134547 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579343+0000", 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T03:40:22.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583029+0000", 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036861540000000001 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583113+0000", 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3370999999999998e-05 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.588186+0000", 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050730269999999999 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.cs2 42/31 ECSubWrite(tid=75, reqid=client.4615.0:256, at_version=42'25, trim_to=0'0, roll_forward_to=41'23))", 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579890+0000", 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79881993699999998, 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001594294, 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.412 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.413 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.413 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579890+0000", 2024-09-16T03:40:22.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579890+0000", 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579890+0000", 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8699999999999999e-07 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579913+0000", 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2772000000000002e-05 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579924+0000", 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1164e-05 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579932+0000", 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1869999999999997e-06 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579994+0000", 2024-09-16T03:40:22.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2170999999999998e-05 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.580004+0000", 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0420000000000008e-06 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581453+0000", 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014492769999999999 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581484+0000", 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1494e-05 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.ds0 42/31 ECSubRead(tid=23, to_read={3:b952e66c:::smithi12138290-29:head=0,118784,0}, subchunks={3:b952e66c:::smithi12138290-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.581806+0000", 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79690372300000001, 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016289869999999999, 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581806+0000", 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581806+0000", 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581806+0000", 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581806+0000", 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581825+0000", 2024-09-16T03:40:22.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8587999999999999e-05 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581834+0000", 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1209999999999996e-06 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582310+0000", 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047653099999999998 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583435+0000", 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011247469999999999 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.cs2 42/31 ECSubWrite(tid=76, reqid=client.4615.0:256, at_version=0'0, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.582062+0000", 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79664766899999995, 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00081929300000000002, 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.421 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582062+0000", 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582062+0000", 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582063+0000", 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2049999999999999e-06 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582075+0000", 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1542999999999999e-05 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582086+0000", 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1337e-05 2024-09-16T03:40:22.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582094+0000", 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6469999999999998e-06 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582148+0000", 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4017999999999997e-05 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582157+0000", 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2639999999999997e-06 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582851+0000", 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00069357100000000003 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582882+0000", 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0707999999999998e-05 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.425 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.ds0 42/31 ECSubRead(tid=24, to_read={3:b952e66c:::smithi12138290-29:head=0,118784,0}, subchunks={3:b952e66c:::smithi12138290-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.582229+0000", 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79648070800000004, 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013308319999999999, 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582229+0000", 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582229+0000", 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582229+0000", 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582229+0000", 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582256+0000", 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7246999999999998e-05 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582268+0000", 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1212e-05 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583439+0000", 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011715790000000001 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583560+0000", 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000120794 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpReadReply(3.ds0 42/31 ECSubReadReply(tid=23, attrs_read=0))", 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.582427+0000", 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79628304999999999, 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011931019999999999, 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582427+0000", 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582427+0000", 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582427+0000", 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9699999999999996e-07 2024-09-16T03:40:22.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582511+0000", 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3708000000000006e-05 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582519+0000", 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0660000000000004e-06 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582523+0000", 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3109999999999999e-06 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583585+0000", 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010611419999999999 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583620+0000", 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5377999999999999e-05 2024-09-16T03:40:22.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.434 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.434 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.434 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpReadReply(3.ds0 42/31 ECSubReadReply(tid=23, attrs_read=0))", 2024-09-16T03:40:22.434 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.582877+0000", 2024-09-16T03:40:22.434 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79583265599999997, 2024-09-16T03:40:22.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00081436500000000005, 2024-09-16T03:40:22.434 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.434 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582877+0000", 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582877+0000", 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582877+0000", 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582877+0000", 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582894+0000", 2024-09-16T03:40:22.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6702e-05 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582900+0000", 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4439999999999996e-06 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583642+0000", 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074206800000000002 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583692+0000", 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9150999999999997e-05 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpReadReply(3.ds0 42/31 ECSubReadReply(tid=24, attrs_read=0))", 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.582902+0000", 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79580783600000005, 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084712100000000005, 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.438 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582902+0000", 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582902+0000", 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582903+0000", 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3600000000000003e-07 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582975+0000", 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2498999999999998e-05 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582985+0000", 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3309999999999993e-06 2024-09-16T03:40:22.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582991+0000", 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0390000000000001e-06 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583714+0000", 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000723337 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583749+0000", 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5278999999999999e-05 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpReadReply(3.ds0 42/31 ECSubReadReply(tid=23, attrs_read=0))", 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.583215+0000", 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79549505300000001, 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00059478600000000003, 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.442 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583215+0000", 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583215+0000", 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583216+0000", 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583295+0000", 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9417000000000002e-05 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583306+0000", 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1107999999999999e-05 2024-09-16T03:40:22.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583319+0000", 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2510999999999999e-05 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583775+0000", 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045625899999999998 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583810+0000", 2024-09-16T03:40:22.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4654999999999999e-05 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.ds0 42/31 ECSubRead(tid=25, to_read={3:b952e66c:::smithi12138290-29:head=0,118784,0}, subchunks={3:b952e66c:::smithi12138290-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.583335+0000", 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79537463900000005, 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00060245299999999997, 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583335+0000", 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583335+0000", 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583335+0000", 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583335+0000", 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583354+0000", 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.836e-05 2024-09-16T03:40:22.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583368+0000", 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4567e-05 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583839+0000", 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000470865 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583938+0000", 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8660999999999996e-05 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpReadReply(3.ds0 42/31 ECSubReadReply(tid=24, attrs_read=0))", 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.583378+0000", 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79533216799999995, 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00064456699999999997, 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.450 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583378+0000", 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583378+0000", 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583378+0000", 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583451+0000", 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3051000000000002e-05 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583460+0000", 2024-09-16T03:40:22.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7309999999999999e-06 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583465+0000", 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2159999999999997e-06 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583978+0000", 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051250999999999998 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.584022+0000", 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4579000000000001e-05 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpReadReply(3.ds0 42/31 ECSubReadReply(tid=23, attrs_read=0))", 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.586907+0000", 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79180339700000002, 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000589103, 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.454 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.586907+0000", 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.586907+0000", 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.586907+0000", 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3600000000000002e-07 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587106+0000", 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019893299999999999 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587144+0000", 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7821000000000003e-05 2024-09-16T03:40:22.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587152+0000", 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7190000000000001e-06 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587193+0000", 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1106999999999998e-05 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587496+0000", 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030258699999999999 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpReadReply(3.ds0 42/31 ECSubReadReply(tid=24, attrs_read=0))", 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.587167+0000", 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79154248000000005, 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00078071200000000003, 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587167+0000", 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587167+0000", 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587168+0000", 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587269+0000", 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100994 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587278+0000", 2024-09-16T03:40:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2750000000000001e-06 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587283+0000", 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.916e-06 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587534+0000", 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025120399999999999 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587948+0000", 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00041397299999999999 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpReadReply(3.ds0 42/31 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.587293+0000", 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.79141671400000002, 2024-09-16T03:40:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00090872200000000005, 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587293+0000", 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587293+0000", 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587294+0000", 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.22e-07 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587387+0000", 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3061999999999998e-05 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587395+0000", 2024-09-16T03:40:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587400+0000", 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6029999999999998e-06 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587968+0000", 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056767899999999997 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.588202+0000", 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023436600000000001 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:255 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.589853+0000", 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.78885691599999996, 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00098297899999999993, 2024-09-16T03:40:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 255 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.589853+0000", 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.589853+0000", 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.589854+0000", 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.437e-06 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.589862+0000", 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.486e-06 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.589874+0000", 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1705999999999999e-05 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.589882+0000", 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1289999999999996e-06 2024-09-16T03:40:22.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.589928+0000", 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6155000000000001e-05 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.589987+0000", 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8631000000000002e-05 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.590836+0000", 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084943499999999995 2024-09-16T03:40:22.470 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.471 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:22.471 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:22.471 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:22.472 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:40:22.473 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:22.471+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:22.550 INFO:teuthology.orchestra.run.smithi076.stderr:set 2.2 pg_upmap mapping to [14,3] 2024-09-16T03:40:22.554 INFO:tasks.rados.rados.0.smithi121.stdout:104: done (2 left) 2024-09-16T03:40:22.554 INFO:tasks.rados.rados.0.smithi121.stdout:105: done (1 left) 2024-09-16T03:40:22.555 INFO:tasks.rados.rados.0.smithi121.stdout:106: done (0 left) 2024-09-16T03:40:22.555 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 24 current snap is 6 2024-09-16T03:40:22.555 INFO:tasks.rados.rados.0.smithi121.stdout:107: seq_num 72 ranges {1622016=442368,2064384=720896,2785280=49152} 2024-09-16T03:40:22.556 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:40:22.566 INFO:tasks.rados.rados.0.smithi121.stdout:107: writing smithi12138290-24 from 1622016 to 2064384 tid 1 2024-09-16T03:40:22.569 INFO:tasks.rados.rados.0.smithi121.stdout:107: writing smithi12138290-24 from 2064384 to 2785280 tid 2 2024-09-16T03:40:22.572 INFO:tasks.rados.rados.0.smithi121.stdout:107: writing smithi12138290-24 from 2785280 to 2834432 tid 3 2024-09-16T03:40:22.573 INFO:tasks.rados.rados.0.smithi121.stdout:108: copy_from oid 14 from oid 10 current snap is 6 2024-09-16T03:40:22.573 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 28 current snap is 6 2024-09-16T03:40:22.573 INFO:tasks.rados.rados.0.smithi121.stdout:109: seq_num 73 ranges {458752=458752,917504=475136,1392640=294912} 2024-09-16T03:40:22.574 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:22.572+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:22.575 INFO:tasks.rados.rados.0.smithi121.stdout:109: writing smithi12138290-28 from 458752 to 917504 tid 1 2024-09-16T03:40:22.578 INFO:tasks.rados.rados.0.smithi121.stdout:109: writing smithi12138290-28 from 917504 to 1392640 tid 2 2024-09-16T03:40:22.580 INFO:tasks.rados.rados.0.smithi121.stdout:109: writing smithi12138290-28 from 1392640 to 1687552 tid 3 2024-09-16T03:40:22.581 INFO:tasks.rados.rados.0.smithi121.stdout:110: snap_create 2024-09-16T03:40:22.581 INFO:tasks.rados.rados.0.smithi121.stdout:107: finishing write tid 1 to smithi12138290-24 2024-09-16T03:40:22.581 INFO:tasks.rados.rados.0.smithi121.stdout:107: finishing write tid 2 to smithi12138290-24 2024-09-16T03:40:22.581 INFO:tasks.rados.rados.0.smithi121.stdout:107: finishing write tid 3 to smithi12138290-24 2024-09-16T03:40:22.581 INFO:tasks.rados.rados.0.smithi121.stdout:107: finishing write tid 4 to smithi12138290-24 2024-09-16T03:40:22.581 INFO:tasks.rados.rados.0.smithi121.stdout:107: finishing write tid 5 to smithi12138290-24 2024-09-16T03:40:22.581 INFO:tasks.rados.rados.0.smithi121.stdout:107: oid 24 updating version 0 to 15 2024-09-16T03:40:22.581 INFO:tasks.rados.rados.0.smithi121.stdout:107: oid 24 updating version 15 to 17 2024-09-16T03:40:22.581 INFO:tasks.rados.rados.0.smithi121.stdout:107: oid 24 updating version 17 to 18 2024-09-16T03:40:22.582 INFO:tasks.rados.rados.0.smithi121.stdout:107: oid 24 version 18 is already newer than 16 2024-09-16T03:40:22.582 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 24 v 18 (ObjNum 72 snap 6 seq_num 72) dirty exists 2024-09-16T03:40:22.582 INFO:tasks.rados.rados.0.smithi121.stdout:107: left oid 24 (ObjNum 72 snap 6 seq_num 72) 2024-09-16T03:40:22.582 INFO:tasks.rados.rados.0.smithi121.stdout:108: finishing copy_from racing read to smithi12138290-14 2024-09-16T03:40:22.582 INFO:tasks.rados.rados.0.smithi121.stdout:108: finishing copy_from to smithi12138290-14 2024-09-16T03:40:22.582 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 14 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:40:22.585 INFO:tasks.rados.rados.0.smithi121.stdout:109: finishing write tid 1 to smithi12138290-28 2024-09-16T03:40:22.585 INFO:tasks.rados.rados.0.smithi121.stdout:109: finishing write tid 2 to smithi12138290-28 2024-09-16T03:40:22.586 INFO:tasks.rados.rados.0.smithi121.stdout:109: finishing write tid 3 to smithi12138290-28 2024-09-16T03:40:22.587 INFO:tasks.rados.rados.0.smithi121.stdout:109: finishing write tid 4 to smithi12138290-28 2024-09-16T03:40:22.589 INFO:tasks.rados.rados.0.smithi121.stdout:109: finishing write tid 5 to smithi12138290-28 2024-09-16T03:40:22.589 INFO:tasks.rados.rados.0.smithi121.stdout:109: oid 28 updating version 0 to 8 2024-09-16T03:40:22.589 INFO:tasks.rados.rados.0.smithi121.stdout:109: oid 28 version 8 is already newer than 6 2024-09-16T03:40:22.589 INFO:tasks.rados.rados.0.smithi121.stdout:109: oid 28 version 8 is already newer than 7 2024-09-16T03:40:22.589 INFO:tasks.rados.rados.0.smithi121.stdout:109: oid 28 updating version 8 to 9 2024-09-16T03:40:22.589 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 28 v 9 (ObjNum 73 snap 6 seq_num 73) dirty exists 2024-09-16T03:40:22.589 INFO:tasks.rados.rados.0.smithi121.stdout:109: left oid 28 (ObjNum 73 snap 6 seq_num 73) 2024-09-16T03:40:22.598 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:22.599 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:22.608 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:40:22.674 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:22.673+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:22.712 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:22.712 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:22.712 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:22.713 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:22.713 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:22.722 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:40:22.776 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:22.775+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:22.827 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:22.827 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 37/31 ECSubWrite(tid=23, reqid=client.4615.0:205, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.525219+0000", 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.2995879940000004, 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003769924, 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.828 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525219+0000", 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525219+0000", 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.29e-07 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525567+0000", 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00034700399999999999 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525577+0000", 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0505e-05 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525583+0000", 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3959999999999997e-06 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525624+0000", 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1437000000000003e-05 2024-09-16T03:40:22.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525630+0000", 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6949999999999997e-06 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.528950+0000", 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033207219999999999 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.528989+0000", 2024-09-16T03:40:22.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9035999999999997e-05 2024-09-16T03:40:22.833 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.833 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.833 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.833 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.cs3 37/31 ECSubWrite(tid=60, reqid=client.4615.0:209, at_version=37'22, trim_to=0'0, roll_forward_to=34'20))", 2024-09-16T03:40:22.833 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.525933+0000", 2024-09-16T03:40:22.834 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.2988749149999999, 2024-09-16T03:40:22.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033302599999999998, 2024-09-16T03:40:22.834 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.834 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.834 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.834 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525933+0000", 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525933+0000", 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525933+0000", 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9200000000000001e-07 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525949+0000", 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5676999999999999e-05 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525955+0000", 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3489999999999997e-06 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.525961+0000", 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5400000000000003e-06 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.526016+0000", 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5359999999999999e-05 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.526024+0000", 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3399999999999998e-06 2024-09-16T03:40:22.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529236+0000", 2024-09-16T03:40:22.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032121020000000001 2024-09-16T03:40:22.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.529263+0000", 2024-09-16T03:40:22.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6400000000000001e-05 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.bs4 37/31 ECSubWrite(tid=28, reqid=client.4615.0:215, at_version=37'14, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.557989+0000", 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.2668180309999997, 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003134689, 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.839 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557989+0000", 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557989+0000", 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.557990+0000", 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9100000000000004e-07 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558004+0000", 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3538e-05 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558013+0000", 2024-09-16T03:40:22.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8109999999999997e-06 2024-09-16T03:40:22.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558018+0000", 2024-09-16T03:40:22.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0730000000000004e-06 2024-09-16T03:40:22.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558065+0000", 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7015000000000003e-05 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.558071+0000", 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4650000000000002e-06 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561094+0000", 2024-09-16T03:40:22.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030226200000000002 2024-09-16T03:40:22.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.561124+0000", 2024-09-16T03:40:22.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0576e-05 2024-09-16T03:40:22.844 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.845 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.845 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.845 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:221 3.0s0 3:0353c118:::smithi12138290-10:3 [read 0~163840,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:22.846 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.573123+0000", 2024-09-16T03:40:22.846 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.2516843289999997, 2024-09-16T03:40:22.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038415390000000002, 2024-09-16T03:40:22.847 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.847 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.847 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:22.847 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:22.848 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:22.848 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:22.848 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 221 2024-09-16T03:40:22.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.848 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573123+0000", 2024-09-16T03:40:22.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573123+0000", 2024-09-16T03:40:22.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573125+0000", 2024-09-16T03:40:22.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4389999999999999e-06 2024-09-16T03:40:22.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573132+0000", 2024-09-16T03:40:22.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7829999999999999e-06 2024-09-16T03:40:22.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573145+0000", 2024-09-16T03:40:22.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2618e-05 2024-09-16T03:40:22.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573151+0000", 2024-09-16T03:40:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1829999999999999e-06 2024-09-16T03:40:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573209+0000", 2024-09-16T03:40:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7766999999999998e-05 2024-09-16T03:40:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573300+0000", 2024-09-16T03:40:22.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0982000000000004e-05 2024-09-16T03:40:22.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.576965+0000", 2024-09-16T03:40:22.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036647670000000002 2024-09-16T03:40:22.857 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.857 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.857 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.858 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:222 3.0s0 3:0353c118:::smithi12138290-10:3 [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:22.858 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.573168+0000", 2024-09-16T03:40:22.858 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.2516390289999997, 2024-09-16T03:40:22.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043920840000000001, 2024-09-16T03:40:22.858 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.859 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.859 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:22.859 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:22.859 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:22.859 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:22.860 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 222 2024-09-16T03:40:22.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.860 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573168+0000", 2024-09-16T03:40:22.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573168+0000", 2024-09-16T03:40:22.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573169+0000", 2024-09-16T03:40:22.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6100000000000002e-07 2024-09-16T03:40:22.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573172+0000", 2024-09-16T03:40:22.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7190000000000001e-06 2024-09-16T03:40:22.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573178+0000", 2024-09-16T03:40:22.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7649999999999996e-06 2024-09-16T03:40:22.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573183+0000", 2024-09-16T03:40:22.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6260000000000003e-06 2024-09-16T03:40:22.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573678+0000", 2024-09-16T03:40:22.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00049521899999999998 2024-09-16T03:40:22.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:22.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573772+0000", 2024-09-16T03:40:22.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4251000000000003e-05 2024-09-16T03:40:22.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.577561+0000", 2024-09-16T03:40:22.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037881429999999999 2024-09-16T03:40:22.868 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.868 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.868 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.869 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:223 3.0s0 3:0353c118:::smithi12138290-10:3 [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:22.869 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.573199+0000", 2024-09-16T03:40:22.869 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.2516084960000002, 2024-09-16T03:40:22.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046986069999999996, 2024-09-16T03:40:22.870 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.870 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.870 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:22.870 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:22.870 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:22.871 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:22.871 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 223 2024-09-16T03:40:22.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.871 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573199+0000", 2024-09-16T03:40:22.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573199+0000", 2024-09-16T03:40:22.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573199+0000", 2024-09-16T03:40:22.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5999999999999999e-07 2024-09-16T03:40:22.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573203+0000", 2024-09-16T03:40:22.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8149999999999999e-06 2024-09-16T03:40:22.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573209+0000", 2024-09-16T03:40:22.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-09-16T03:40:22.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573214+0000", 2024-09-16T03:40:22.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6240000000000001e-06 2024-09-16T03:40:22.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.577069+0000", 2024-09-16T03:40:22.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00385506 2024-09-16T03:40:22.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:22.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.577118+0000", 2024-09-16T03:40:22.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9670999999999999e-05 2024-09-16T03:40:22.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.577898+0000", 2024-09-16T03:40:22.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077912399999999996 2024-09-16T03:40:22.879 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.879 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.880 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.880 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpRead(3.0s0 37/31 ECSubRead(tid=17, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:22.880 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.573527+0000", 2024-09-16T03:40:22.881 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.2512808809999996, 2024-09-16T03:40:22.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030623949999999999, 2024-09-16T03:40:22.881 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.881 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.881 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.881 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573527+0000", 2024-09-16T03:40:22.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573527+0000", 2024-09-16T03:40:22.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573527+0000", 2024-09-16T03:40:22.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573527+0000", 2024-09-16T03:40:22.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573539+0000", 2024-09-16T03:40:22.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2601e-05 2024-09-16T03:40:22.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573552+0000", 2024-09-16T03:40:22.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2325e-05 2024-09-16T03:40:22.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573954+0000", 2024-09-16T03:40:22.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040221300000000001 2024-09-16T03:40:22.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.576589+0000", 2024-09-16T03:40:22.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026352559999999999 2024-09-16T03:40:22.889 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.889 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.889 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpRead(3.0s0 37/31 ECSubRead(tid=18, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.573873+0000", 2024-09-16T03:40:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.2509347960000001, 2024-09-16T03:40:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027940270000000001, 2024-09-16T03:40:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573873+0000", 2024-09-16T03:40:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573873+0000", 2024-09-16T03:40:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573873+0000", 2024-09-16T03:40:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573873+0000", 2024-09-16T03:40:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573889+0000", 2024-09-16T03:40:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5905000000000001e-05 2024-09-16T03:40:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.573897+0000", 2024-09-16T03:40:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7800000000000006e-06 2024-09-16T03:40:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.576618+0000", 2024-09-16T03:40:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027203330000000001 2024-09-16T03:40:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.576667+0000", 2024-09-16T03:40:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9008999999999998e-05 2024-09-16T03:40:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpReadReply(3.0s0 37/31 ECSubReadReply(tid=18, attrs_read=0))", 2024-09-16T03:40:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.574309+0000", 2024-09-16T03:40:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.250498984, 2024-09-16T03:40:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002736002, 2024-09-16T03:40:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574309+0000", 2024-09-16T03:40:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574309+0000", 2024-09-16T03:40:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574309+0000", 2024-09-16T03:40:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3799999999999998e-07 2024-09-16T03:40:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574338+0000", 2024-09-16T03:40:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9022000000000001e-05 2024-09-16T03:40:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574345+0000", 2024-09-16T03:40:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4540000000000001e-06 2024-09-16T03:40:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574349+0000", 2024-09-16T03:40:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8929999999999998e-06 2024-09-16T03:40:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.577006+0000", 2024-09-16T03:40:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026568609999999999 2024-09-16T03:40:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.577045+0000", 2024-09-16T03:40:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8433999999999998e-05 2024-09-16T03:40:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpReadReply(3.0s0 37/31 ECSubReadReply(tid=18, attrs_read=0))", 2024-09-16T03:40:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.574386+0000", 2024-09-16T03:40:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.2504211319999996, 2024-09-16T03:40:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029152610000000002, 2024-09-16T03:40:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574386+0000", 2024-09-16T03:40:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574386+0000", 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574387+0000", 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5400000000000002e-07 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574419+0000", 2024-09-16T03:40:22.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2351999999999998e-05 2024-09-16T03:40:22.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574425+0000", 2024-09-16T03:40:22.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.626e-06 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.574428+0000", 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1599999999999998e-06 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.577275+0000", 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028469229999999999 2024-09-16T03:40:22.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.577302+0000", 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6945999999999999e-05 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs5 38/31 ECSubWrite(tid=63, reqid=client.4615.0:226, at_version=38'10, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:22.914 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.544170+0000", 2024-09-16T03:40:22.915 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.280637187, 2024-09-16T03:40:22.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056030519999999999, 2024-09-16T03:40:22.915 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.915 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.915 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.915 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544170+0000", 2024-09-16T03:40:22.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544170+0000", 2024-09-16T03:40:22.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544171+0000", 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4300000000000003e-07 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544310+0000", 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013949599999999999 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544330+0000", 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9539e-05 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544342+0000", 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.155e-05 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544395+0000", 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3318e-05 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544404+0000", 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.149e-06 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549741+0000", 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005337456 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549773+0000", 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1900999999999998e-05 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs5 38/31 ECSubWrite(tid=64, reqid=client.4615.0:227, at_version=38'11, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.544765+0000", 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.2800420819999996, 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050451089999999999, 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544765+0000", 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544765+0000", 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544766+0000", 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4000000000000001e-07 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544781+0000", 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5073e-05 2024-09-16T03:40:22.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544792+0000", 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0665e-05 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544800+0000", 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8769999999999993e-06 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544855+0000", 2024-09-16T03:40:22.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4923000000000003e-05 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544863+0000", 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4079999999999998e-06 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549792+0000", 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049294450000000002 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549811+0000", 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8077999999999999e-05 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs5 42/31 ECSubWrite(tid=73, reqid=osd.0.0:73, at_version=42'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.555619+0000", 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.2691886139999999, 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046244220000000004, 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.924 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555619+0000", 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555619+0000", 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555619+0000", 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0399999999999996e-07 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555649+0000", 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9558999999999999e-05 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555659+0000", 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0144e-05 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555668+0000", 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0589999999999992e-06 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555725+0000", 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6626999999999998e-05 2024-09-16T03:40:22.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555732+0000", 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.742e-06 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560171+0000", 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044394090000000001 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560243+0000", 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2278000000000002e-05 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:251 3.0s0 3:00cc7e13:::smithi12138290-42:head [copy-from ver 4] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.2680716869999999, 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013120588000000001, 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:22.929 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 251 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:40:22.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556738+0000", 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8989999999999999e-06 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556745+0000", 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556759+0000", 2024-09-16T03:40:22.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.451e-05 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556766+0000", 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1260000000000003e-06 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556817+0000", 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1693999999999997e-05 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556824+0000", 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8549999999999996e-06 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557940+0000", 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001115888 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558009+0000", 2024-09-16T03:40:22.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8707999999999993e-05 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564426+0000", 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064170659999999999 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.566682+0000", 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022565039999999999 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569779+0000", 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030962770000000001 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569856+0000", 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7843999999999995e-05 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.936 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:252 3.0s0 3:00cc7e13:::smithi12138290-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:22.936 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.556781+0000", 2024-09-16T03:40:22.936 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.2680265560000001, 2024-09-16T03:40:22.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008452701, 2024-09-16T03:40:22.936 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.936 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.936 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:22.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:22.937 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:22.937 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:22.937 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 252 2024-09-16T03:40:22.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.937 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556781+0000", 2024-09-16T03:40:22.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556781+0000", 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556782+0000", 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8899999999999999e-07 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556786+0000", 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0910000000000003e-06 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556795+0000", 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1249999999999999e-06 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556801+0000", 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1040000000000003e-06 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556876+0000", 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5113999999999999e-05 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556880+0000", 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2749999999999997e-06 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558172+0000", 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001291936 2024-09-16T03:40:22.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558261+0000", 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8628999999999995e-05 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.565110+0000", 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068490360000000002 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.565168+0000", 2024-09-16T03:40:22.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8087999999999997e-05 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.565234+0000", 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5714000000000001e-05 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 42/31 ECSubWrite(tid=30, reqid=osd.14.0:30, at_version=42'20, trim_to=0'0, roll_forward_to=41'18))", 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.557778+0000", 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.2670294440000001, 2024-09-16T03:40:22.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047408789999999999, 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557778+0000", 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557778+0000", 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557778+0000", 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0999999999999994e-08 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557796+0000", 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8079000000000001e-05 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557805+0000", 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6149999999999997e-06 2024-09-16T03:40:22.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557812+0000", 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1400000000000002e-06 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557873+0000", 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557882+0000", 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7269999999999996e-06 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562477+0000", 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045955409999999999 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562519+0000", 2024-09-16T03:40:22.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1545000000000003e-05 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.8s2 42/31 ECSubWrite(tid=183, reqid=osd.11.0:182, at_version=0'0, trim_to=0'0, roll_forward_to=42'8))", 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.560183+0000", 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.2646240739999999, 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003453373, 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.947 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560183+0000", 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560183+0000", 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560185+0000", 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7600000000000001e-06 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560212+0000", 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6855999999999998e-05 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560244+0000", 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1677000000000002e-05 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560253+0000", 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2e-06 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560320+0000", 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7403999999999994e-05 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560329+0000", 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6270000000000006e-06 2024-09-16T03:40:22.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.563599+0000", 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032700849999999998 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.563637+0000", 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7764e-05 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.951 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs5 42/31 ECSubWrite(tid=74, reqid=osd.0.0:73, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.562860+0000", 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.261947454, 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035098870000000002, 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562860+0000", 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562860+0000", 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562861+0000", 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0210000000000001e-06 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562871+0000", 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0096e-05 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562887+0000", 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6215e-05 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562894+0000", 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6560000000000003e-06 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562964+0000", 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9609000000000004e-05 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562976+0000", 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.19e-05 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.566330+0000", 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033546180000000002 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.566370+0000", 2024-09-16T03:40:22.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9771999999999998e-05 2024-09-16T03:40:22.956 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.956 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.956 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.956 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 42/31 ECSubWrite(tid=32, reqid=osd.14.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:22.956 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.563750+0000", 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.2610574809999999, 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029326090000000001, 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.563750+0000", 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.563750+0000", 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.563751+0000", 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.563760+0000", 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6479999999999998e-06 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.563771+0000", 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0345e-05 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.563779+0000", 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1159999999999999e-06 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.563834+0000", 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5176999999999997e-05 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.563841+0000", 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6930000000000001e-06 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.566653+0000", 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028123390000000001 2024-09-16T03:40:22.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.566683+0000", 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9567e-05 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s5 42/31 ECSubWrite(tid=24, reqid=client.4615.0:268, at_version=42'15, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.629041+0000", 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.195766055, 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025224560000000002, 2024-09-16T03:40:22.961 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.629041+0000", 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.629041+0000", 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.629042+0000", 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3200000000000005e-07 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.629118+0000", 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6038000000000006e-05 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.629128+0000", 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7869999999999996e-06 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.629136+0000", 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7100000000000007e-06 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.629185+0000", 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9586999999999998e-05 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.629192+0000", 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1790000000000002e-06 2024-09-16T03:40:22.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.631526+0000", 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023333379999999999 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.631564+0000", 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8284999999999997e-05 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:22.965 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:22.967 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:40:22.968 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:22.965+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:23.067 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:23.065+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:23.111 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:23.112 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:23.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:40:23.169 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:23.166+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:23.223 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:23.223 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:23.223 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:23.223 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:23.223 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:23.231 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:40:23.254 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:23.275 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:23.274+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:23.336 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:23.336 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:23.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:23.336 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:23.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.337 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:20 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'10, mlcod=25'8)", 2024-09-16T03:40:23.337 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:58.296291+0000", 2024-09-16T03:40:23.337 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.036496145000001, 2024-09-16T03:40:23.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00404033, 2024-09-16T03:40:23.337 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.337 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.338 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.338 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296291+0000", 2024-09-16T03:40:23.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296291+0000", 2024-09-16T03:40:23.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296291+0000", 2024-09-16T03:40:23.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0500000000000003e-07 2024-09-16T03:40:23.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296373+0000", 2024-09-16T03:40:23.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1823999999999995e-05 2024-09-16T03:40:23.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296386+0000", 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2747e-05 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296398+0000", 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2218999999999999e-05 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296456+0000", 2024-09-16T03:40:23.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7361999999999997e-05 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.296471+0000", 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5829999999999999e-05 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.300291+0000", 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038192090000000001 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.300331+0000", 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0433999999999999e-05 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:39 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'25, mlcod=25'22)", 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.014830719999999, 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042816670000000003, 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.343 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317957+0000", 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.54e-07 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318024+0000", 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6730000000000007e-05 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318033+0000", 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1589999999999999e-06 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318038+0000", 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8040000000000001e-06 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318080+0000", 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2475000000000002e-05 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318093+0000", 2024-09-16T03:40:23.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3315e-05 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.322203+0000", 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041094620000000004 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.322238+0000", 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4968000000000001e-05 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:164 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'118, mlcod=25'113)", 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.174553585000002, 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047753090000000002, 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:23.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158234+0000", 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8100000000000003e-07 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158294+0000", 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0359999999999998e-05 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158301+0000", 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3159999999999999e-06 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158305+0000", 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3900000000000002e-06 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158343+0000", 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7793000000000002e-05 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158361+0000", 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8230999999999999e-05 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.162979+0000", 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046178230000000001 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.163009+0000", 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9915000000000001e-05 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:178 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'128, mlcod=25'125)", 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.049148721000002, 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058842529999999999, 2024-09-16T03:40:23.352 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283639+0000", 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283643+0000", 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3980000000000001e-06 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283652+0000", 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6430000000000001e-06 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283656+0000", 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7339999999999999e-06 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283698+0000", 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1785000000000001e-05 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283715+0000", 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6555e-05 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289498+0000", 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0057832869999999998 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289522+0000", 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4290999999999998e-05 2024-09-16T03:40:23.356 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:179 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'129, mlcod=25'125)", 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.048677592000001, 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066372109999999996, 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284110+0000", 2024-09-16T03:40:23.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.89e-07 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284175+0000", 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5667000000000007e-05 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284186+0000", 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.009e-05 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284190+0000", 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7600000000000002e-06 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284222+0000", 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1888999999999999e-05 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284234+0000", 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.225e-05 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290722+0000", 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064871909999999998 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290746+0000", 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4875000000000002e-05 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:180 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'130, mlcod=25'125)", 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.048063283000001, 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072760109999999998, 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.68e-07 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284729+0000", 2024-09-16T03:40:23.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4009999999999999e-06 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284738+0000", 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6940000000000007e-06 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284743+0000", 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5549999999999996e-06 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286547+0000", 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0018044770000000001 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286567+0000", 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9272000000000001e-05 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.291958+0000", 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053911540000000004 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292000+0000", 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.189e-05 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:181 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'131, mlcod=25'126)", 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.047522733000001, 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067648710000000004, 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.366 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285265+0000", 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6100000000000001e-07 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285269+0000", 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5020000000000003e-06 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285279+0000", 2024-09-16T03:40:23.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6879999999999994e-06 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285286+0000", 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4590000000000001e-06 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286842+0000", 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015560000000000001 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286854+0000", 2024-09-16T03:40:23.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1959e-05 2024-09-16T03:40:23.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292001+0000", 2024-09-16T03:40:23.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051468859999999998 2024-09-16T03:40:23.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292029+0000", 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7815999999999999e-05 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:182 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'132, mlcod=25'126)", 2024-09-16T03:40:23.371 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.047078013, 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0063484680000000003, 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2200000000000004e-07 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285737+0000", 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7886999999999999e-05 2024-09-16T03:40:23.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285747+0000", 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9459999999999993e-06 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285752+0000", 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287024+0000", 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012715409999999999 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287042+0000", 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8919999999999998e-05 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292031+0000", 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049888179999999999 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292057+0000", 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6057000000000001e-05 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:183 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'133, mlcod=25'126)", 2024-09-16T03:40:23.376 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:23.377 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.046382918999999, 2024-09-16T03:40:23.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060351210000000001, 2024-09-16T03:40:23.377 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.377 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.377 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.377 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286405+0000", 2024-09-16T03:40:23.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4300000000000001e-06 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286429+0000", 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3459999999999999e-05 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286439+0000", 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0105999999999999e-05 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286443+0000", 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.527e-06 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289328+0000", 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028841940000000001 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289340+0000", 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1861e-05 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292411+0000", 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030717399999999999 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292439+0000", 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7803000000000001e-05 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:184 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'134, mlcod=25'126)", 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.045968552000001, 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056342509999999998, 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286819+0000", 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5499999999999998e-07 2024-09-16T03:40:23.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286823+0000", 2024-09-16T03:40:23.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2889999999999999e-06 2024-09-16T03:40:23.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286833+0000", 2024-09-16T03:40:23.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4769999999999992e-06 2024-09-16T03:40:23.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286839+0000", 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6599999999999998e-06 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289572+0000", 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002733161 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289593+0000", 2024-09-16T03:40:23.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0494999999999999e-05 2024-09-16T03:40:23.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292440+0000", 2024-09-16T03:40:23.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028470660000000001 2024-09-16T03:40:23.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292453+0000", 2024-09-16T03:40:23.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2548000000000001e-05 2024-09-16T03:40:23.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.390 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:185 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'135, mlcod=25'126)", 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.0454048, 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050824039999999996, 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:23.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287383+0000", 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287410+0000", 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6985e-05 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287419+0000", 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6879999999999994e-06 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287424+0000", 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6829999999999999e-06 2024-09-16T03:40:23.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289884+0000", 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002459639 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289917+0000", 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3303999999999998e-05 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292453+0000", 2024-09-16T03:40:23.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025360769999999999 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292464+0000", 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1364e-05 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:186 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'136, mlcod=25'126)", 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.044890167999998, 2024-09-16T03:40:23.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045805179999999996, 2024-09-16T03:40:23.396 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.396 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.396 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.396 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:23.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3200000000000005e-07 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287902+0000", 2024-09-16T03:40:23.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.634e-06 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287912+0000", 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7739999999999999e-06 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287916+0000", 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6099999999999999e-06 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290274+0000", 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023574519999999999 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290289+0000", 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5481e-05 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292466+0000", 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021765439999999999 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292477+0000", 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:187 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'137, mlcod=25'126)", 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.288323+0000", 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.044463729, 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041669410000000004, 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.400 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288323+0000", 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288323+0000", 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288324+0000", 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3300000000000002e-07 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288328+0000", 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4689999999999999e-06 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288338+0000", 2024-09-16T03:40:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8169999999999994e-06 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.288343+0000", 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9370000000000003e-06 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290524+0000", 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021806009999999999 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290536+0000", 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2751e-05 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292478+0000", 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001941454 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292490+0000", 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2279000000000001e-05 2024-09-16T03:40:23.404 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:204 1.0 e26/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'150, mlcod=25'145)", 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.921521997999999, 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051027959999999997, 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.405 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411272+0000", 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1029999999999998e-06 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411280+0000", 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3129999999999999e-06 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411287+0000", 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2570000000000001e-06 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411347+0000", 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0214000000000003e-05 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411361+0000", 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3735e-05 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416329+0000", 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049679889999999999 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416368+0000", 2024-09-16T03:40:23.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8634999999999999e-05 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:205 1.0 e26/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'151, mlcod=26'147)", 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.920982729999999, 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004582226, 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.410 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411805+0000", 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2699999999999999e-07 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411808+0000", 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6710000000000001e-06 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411815+0000", 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9480000000000002e-06 2024-09-16T03:40:23.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411820+0000", 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0270000000000003e-06 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411851+0000", 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0474e-05 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411862+0000", 2024-09-16T03:40:23.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.171e-05 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416370+0000", 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045072810000000001 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416386+0000", 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6588000000000001e-05 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:206 1.0 e26/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'152, mlcod=26'147)", 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:00.412162+0000", 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.920624898, 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042383129999999996, 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412162+0000", 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412162+0000", 2024-09-16T03:40:23.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412162+0000", 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9299999999999998e-07 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412240+0000", 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7395e-05 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412247+0000", 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.294e-06 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412252+0000", 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8600000000000001e-06 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412285+0000", 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2843999999999999e-05 2024-09-16T03:40:23.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.412298+0000", 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3159000000000001e-05 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416387+0000", 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040889890000000003 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416400+0000", 2024-09-16T03:40:23.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3278999999999999e-05 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es2 37/31 ECSubWrite(tid=26, reqid=client.4615.0:218, at_version=37'8, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.556575+0000", 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.7762115739999995, 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044039439999999999, 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:23.420 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556575+0000", 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556575+0000", 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556578+0000", 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8490000000000002e-06 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556597+0000", 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8513999999999999e-05 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556607+0000", 2024-09-16T03:40:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0309000000000001e-05 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556616+0000", 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3889999999999994e-06 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556677+0000", 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1066000000000001e-05 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:23.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556686+0000", 2024-09-16T03:40:23.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.952e-06 2024-09-16T03:40:23.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:23.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560938+0000", 2024-09-16T03:40:23.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042516860000000002 2024-09-16T03:40:23.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560979+0000", 2024-09-16T03:40:23.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1179e-05 2024-09-16T03:40:23.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.425 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.425 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.5s6 42/31 ECSubWrite(tid=23, reqid=osd.8.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:23.426 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.555040+0000", 2024-09-16T03:40:23.426 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.7777471490000001, 2024-09-16T03:40:23.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005476687, 2024-09-16T03:40:23.426 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.426 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.426 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:23.427 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555040+0000", 2024-09-16T03:40:23.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555040+0000", 2024-09-16T03:40:23.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555041+0000", 2024-09-16T03:40:23.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0500000000000002e-07 2024-09-16T03:40:23.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555051+0000", 2024-09-16T03:40:23.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0305e-05 2024-09-16T03:40:23.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555063+0000", 2024-09-16T03:40:23.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2002e-05 2024-09-16T03:40:23.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555069+0000", 2024-09-16T03:40:23.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5889999999999999e-06 2024-09-16T03:40:23.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555132+0000", 2024-09-16T03:40:23.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2514000000000005e-05 2024-09-16T03:40:23.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:23.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555142+0000", 2024-09-16T03:40:23.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0118999999999999e-05 2024-09-16T03:40:23.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:23.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560482+0000", 2024-09-16T03:40:23.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053403239999999996 2024-09-16T03:40:23.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560516+0000", 2024-09-16T03:40:23.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3929e-05 2024-09-16T03:40:23.435 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.435 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.436 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.436 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 42/31 ECSubWrite(tid=73, reqid=osd.0.0:73, at_version=42'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:23.436 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:23.437 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.7768678009999999, 2024-09-16T03:40:23.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045524069999999996, 2024-09-16T03:40:23.437 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.437 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.437 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:23.437 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:23.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:23.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:23.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7700000000000001e-07 2024-09-16T03:40:23.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555946+0000", 2024-09-16T03:40:23.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6560999999999999e-05 2024-09-16T03:40:23.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555959+0000", 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3042e-05 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555968+0000", 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1079999999999999e-06 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556053+0000", 2024-09-16T03:40:23.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4975999999999994e-05 2024-09-16T03:40:23.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:23.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556063+0000", 2024-09-16T03:40:23.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6120000000000005e-06 2024-09-16T03:40:23.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:23.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560429+0000", 2024-09-16T03:40:23.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043663369999999996 2024-09-16T03:40:23.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560471+0000", 2024-09-16T03:40:23.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2493999999999997e-05 2024-09-16T03:40:23.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.445 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.445 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 42/31 ECSubWrite(tid=74, reqid=osd.0.0:73, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:23.445 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:40:23.445 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.7699124159999999, 2024-09-16T03:40:23.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064234579999999999, 2024-09-16T03:40:23.446 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:23.446 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:23.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562875+0000", 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562885+0000", 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0069e-05 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562897+0000", 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1619e-05 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562903+0000", 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.652e-06 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562965+0000", 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1576999999999997e-05 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562975+0000", 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0314e-05 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569190+0000", 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062148569999999998 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:23.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:23.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:23.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569298+0000", 2024-09-16T03:40:23.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010764600000000001 2024-09-16T03:40:23.451 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.451 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.451 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.451 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:23.451 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:23.451 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:23.453 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:23.453 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:23.454 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:23.454 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:23.454 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:40:23.455 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:23.455 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:23.456 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:23.456 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:23.456 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:23.456 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:23.457 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:23.450+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:23.460 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:23.460 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:23.477 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:23.477 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:23.494 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:23.494 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:23.510 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:23.510 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:23.526 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:23.527 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:23.544 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:23.553 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:23.551+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:23.560 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T03:40:23.561 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:23.561 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:23.563 INFO:tasks.rados.rados.0.smithi121.stdout:107: done (3 left) 2024-09-16T03:40:23.563 INFO:tasks.rados.rados.0.smithi121.stdout:108: done (2 left) 2024-09-16T03:40:23.563 INFO:tasks.rados.rados.0.smithi121.stdout:109: done (1 left) 2024-09-16T03:40:23.563 INFO:tasks.rados.rados.0.smithi121.stdout:110: done (0 left) 2024-09-16T03:40:23.563 INFO:tasks.rados.rados.0.smithi121.stdout:111: snap_create 2024-09-16T03:40:23.579 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:23.579 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:23.597 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:23.597 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:23.601 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:23.601 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:23.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:40:23.614 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:23.614 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:23.631 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:23.632 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:23.654 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:23.652+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:23.712 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:23.712 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:23.720 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:40:23.756 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:23.754+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:23.823 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:23.823 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:23.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:40:23.856 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:23.854+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:23.934 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:23.934 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:23.942 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:40:23.957 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:23.955+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:24.045 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:24.045 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:24.054 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:40:24.058 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:24.056+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:24.156 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:24.156 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:24.159 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:24.156+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:24.164 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:40:24.260 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:24.258+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:24.280 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:24.280 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:24.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:40:24.361 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:24.359+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:24.394 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:24.394 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:24.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:40:24.461 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:24.460+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:24.547 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:24.547 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:24.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:40:24.562 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:24.560+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:24.567 INFO:tasks.rados.rados.0.smithi121.stdout:111: done (0 left) 2024-09-16T03:40:24.567 INFO:tasks.rados.rados.0.smithi121.stdout:112: snap_create 2024-09-16T03:40:24.659 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:24.659 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:24.662 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:24.660+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:24.668 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:40:24.763 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:24.762+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:24.811 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:24.811 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:24.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:40:24.866 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:24.864+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:24.925 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:24.925 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:24.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:40:24.975 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:24.973+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:25.066 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:25.064+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:25.077 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:25.078 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:25.087 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:40:25.167 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:25.165+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:25.190 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:25.190 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:25.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:40:25.268 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:25.267+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:25.342 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:25.342 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:25.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:40:25.369 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:25.368+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:25.453 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:25.454 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:25.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:40:25.507 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:25.505+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:25.567 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:25.567 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:25.570 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:25.568+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:25.574 INFO:tasks.rados.rados.0.smithi121.stdout:112: done (0 left) 2024-09-16T03:40:25.574 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 4 current snap is 9 2024-09-16T03:40:25.574 INFO:tasks.rados.rados.0.smithi121.stdout:113: seq_num 74 ranges {770048=475136,1245184=622592,1867776=147456} 2024-09-16T03:40:25.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:40:25.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:40:25.581 INFO:tasks.rados.rados.0.smithi121.stdout:113: writing smithi12138290-4 from 770048 to 1245184 tid 1 2024-09-16T03:40:25.584 INFO:tasks.rados.rados.0.smithi121.stdout:113: writing smithi12138290-4 from 1245184 to 1867776 tid 2 2024-09-16T03:40:25.585 INFO:tasks.rados.rados.0.smithi121.stdout:113: writing smithi12138290-4 from 1867776 to 2015232 tid 3 2024-09-16T03:40:25.585 INFO:tasks.rados.rados.0.smithi121.stdout:114: snap_create 2024-09-16T03:40:25.591 INFO:tasks.rados.rados.0.smithi121.stdout:113: finishing write tid 1 to smithi12138290-4 2024-09-16T03:40:25.592 INFO:tasks.rados.rados.0.smithi121.stdout:113: finishing write tid 2 to smithi12138290-4 2024-09-16T03:40:25.593 INFO:tasks.rados.rados.0.smithi121.stdout:113: finishing write tid 3 to smithi12138290-4 2024-09-16T03:40:25.593 INFO:tasks.rados.rados.0.smithi121.stdout:113: finishing write tid 4 to smithi12138290-4 2024-09-16T03:40:25.595 INFO:tasks.rados.rados.0.smithi121.stdout:113: finishing write tid 5 to smithi12138290-4 2024-09-16T03:40:25.595 INFO:tasks.rados.rados.0.smithi121.stdout:113: oid 4 updating version 0 to 16 2024-09-16T03:40:25.595 INFO:tasks.rados.rados.0.smithi121.stdout:113: oid 4 version 16 is already newer than 15 2024-09-16T03:40:25.595 INFO:tasks.rados.rados.0.smithi121.stdout:113: oid 4 updating version 16 to 17 2024-09-16T03:40:25.595 INFO:tasks.rados.rados.0.smithi121.stdout:113: oid 4 updating version 17 to 18 2024-09-16T03:40:25.595 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 4 v 18 (ObjNum 74 snap 9 seq_num 74) dirty exists 2024-09-16T03:40:25.595 INFO:tasks.rados.rados.0.smithi121.stdout:113: left oid 4 (ObjNum 74 snap 9 seq_num 74) 2024-09-16T03:40:25.643 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:25.687 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:25.684+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:25.721 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:25.721 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:25.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:40:25.773 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:25.771+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:25.773 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:25.773 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:25.791 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:25.808 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:25.825 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:25.825 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:25.841 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:25.841 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:25.846 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:25.846 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:25.846 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:25.846 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:25.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:40:25.858 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:25.858 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:25.874 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:25.872+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:25.874 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:25.874 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:25.890 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:25.891 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:25.907 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:25.923 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:25.924 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:25.940 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:25.940 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:25.956 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:25.957 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:25.958 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:25.958 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:25.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:40:25.972 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:25.972 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:25.974 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:25.973+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:25.988 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:25.989 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:26.004 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:26.005 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:26.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:26.003+0000 7f38e5b9e640 0 -- 172.21.15.76:0/2475059295 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f38e0220ce0 msgr2=0x7f38e02230d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:26.072 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:26.072 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:26.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:26.072 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:26.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.072 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpRead(3.0s3 37/31 ECSubRead(tid=17, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:26.072 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.573588+0000", 2024-09-16T03:40:26.072 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.4959707790000003, 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051434400000000002, 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573588+0000", 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573588+0000", 2024-09-16T03:40:26.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573588+0000", 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.11e-07 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573593+0000", 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4999999999999999e-06 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573611+0000", 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7941999999999999e-05 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573620+0000", 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9989999999999997e-06 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573692+0000", 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1734000000000001e-05 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.574102+0000", 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041005799999999998 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpRead(3.0s3 37/31 ECSubRead(tid=18, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.573877+0000", 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.4956817989999998, 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032719999999999998, 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.076 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573877+0000", 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573877+0000", 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573877+0000", 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0500000000000003e-07 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573883+0000", 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.285e-06 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573895+0000", 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1144999999999999e-05 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.573901+0000", 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1990000000000003e-06 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.574128+0000", 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000227696 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.574204+0000", 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5270000000000003e-05 2024-09-16T03:40:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpRead(3.0s3 37/31 ECSubRead(tid=19, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.577209+0000", 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.4923494640000001, 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140949, 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.577209+0000", 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.577209+0000", 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.577209+0000", 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1399999999999997e-07 2024-09-16T03:40:26.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.577215+0000", 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9050000000000002e-06 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.577226+0000", 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0706e-05 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.577235+0000", 2024-09-16T03:40:26.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7369999999999995e-06 2024-09-16T03:40:26.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.577284+0000", 2024-09-16T03:40:26.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9669999999999997e-05 2024-09-16T03:40:26.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.577350+0000", 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5417000000000001e-05 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 42/31 ECSubWrite(tid=20, reqid=client.4615.0:251, at_version=42'11, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.564293+0000", 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.5052654219999999, 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044757829999999997, 2024-09-16T03:40:26.084 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564293+0000", 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564293+0000", 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564293+0000", 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.53e-07 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564405+0000", 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111525 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564417+0000", 2024-09-16T03:40:26.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2046e-05 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564424+0000", 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2760000000000003e-06 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564521+0000", 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7120999999999994e-05 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564532+0000", 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0419e-05 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.568719+0000", 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041873969999999998 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.568769+0000", 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9746000000000001e-05 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.088 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 42/31 ECSubWrite(tid=21, reqid=client.4615.0:251, at_version=0'0, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.569961+0000", 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4995976449999997, 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020021219999999998, 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.569961+0000", 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.569961+0000", 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.569961+0000", 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0200000000000001e-07 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.569972+0000", 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.569983+0000", 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1313e-05 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.569989+0000", 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0800000000000002e-06 2024-09-16T03:40:26.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.570048+0000", 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8486000000000002e-05 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.570058+0000", 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8309999999999996e-06 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571933+0000", 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001875441 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571963+0000", 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9756000000000001e-05 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 42/31 ECSubWrite(tid=22, reqid=client.4615.0:266, at_version=42'13, trim_to=0'0, roll_forward_to=34'11))", 2024-09-16T03:40:26.093 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.624549+0000", 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4450088809999997, 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015427489999999999, 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.624549+0000", 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.624549+0000", 2024-09-16T03:40:26.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.624550+0000", 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4999999999999999e-07 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.624866+0000", 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031661599999999998 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.624878+0000", 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2162e-05 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.624885+0000", 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1010000000000003e-06 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.624953+0000", 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7336999999999996e-05 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.624965+0000", 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2458e-05 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.626052+0000", 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010871769999999999 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.626092+0000", 2024-09-16T03:40:26.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9748e-05 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 42/31 ECSubWrite(tid=23, reqid=client.4615.0:267, at_version=42'14, trim_to=0'0, roll_forward_to=34'11))", 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.627860+0000", 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.441698529, 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012752270000000001, 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.627860+0000", 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.627860+0000", 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.627861+0000", 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4199999999999995e-07 2024-09-16T03:40:26.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628242+0000", 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000381954 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628253+0000", 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0329000000000001e-05 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628259+0000", 2024-09-16T03:40:26.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7080000000000002e-06 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628314+0000", 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5401e-05 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628324+0000", 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0369e-05 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629103+0000", 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00077827199999999999 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629135+0000", 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2452e-05 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 42/31 ECSubWrite(tid=24, reqid=client.4615.0:268, at_version=42'15, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.629096+0000", 2024-09-16T03:40:26.102 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4404623980000002, 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027011040000000002, 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629096+0000", 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629096+0000", 2024-09-16T03:40:26.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629097+0000", 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5700000000000002e-07 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629153+0000", 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5983000000000003e-05 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629163+0000", 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0135e-05 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629171+0000", 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1210000000000007e-06 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629247+0000", 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6074000000000004e-05 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629257+0000", 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0098000000000001e-05 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631773+0000", 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025155759999999998 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631797+0000", 2024-09-16T03:40:26.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.446e-05 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 42/31 ECSubWrite(tid=25, reqid=client.4615.0:269, at_version=42'16, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.629979+0000", 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4395789629999998, 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045141219999999998, 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629979+0000", 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629979+0000", 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629980+0000", 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.95e-07 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629998+0000", 2024-09-16T03:40:26.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7475000000000001e-05 2024-09-16T03:40:26.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630012+0000", 2024-09-16T03:40:26.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3737000000000001e-05 2024-09-16T03:40:26.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630020+0000", 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4239999999999993e-06 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630068+0000", 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7729000000000003e-05 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630073+0000", 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5090000000000001e-06 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634467+0000", 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043935040000000003 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634493+0000", 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6749000000000001e-05 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.111 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 42/31 ECSubWrite(tid=26, reqid=client.4615.0:269, at_version=0'0, trim_to=0'0, roll_forward_to=42'16))", 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.635260+0000", 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4342980379999997, 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00063181499999999998, 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635260+0000", 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635260+0000", 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635261+0000", 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6100000000000001e-07 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635270+0000", 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8710000000000003e-06 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635279+0000", 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3700000000000001e-06 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635288+0000", 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4270000000000008e-06 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635345+0000", 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6981000000000002e-05 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635359+0000", 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4042e-05 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635869+0000", 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051011000000000003 2024-09-16T03:40:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635892+0000", 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3453000000000001e-05 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpRead(3.1s2 42/31 ECSubRead(tid=27, to_read={3:82a78c8d:::smithi12138290-1:head=0,4096,0}, subchunks={3:82a78c8d:::smithi12138290-1:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.635601+0000", 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4339569790000004, 2024-09-16T03:40:26.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016969700000000001, 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635601+0000", 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635601+0000", 2024-09-16T03:40:26.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635602+0000", 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1399999999999997e-07 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635608+0000", 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635619+0000", 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0733999999999999e-05 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635625+0000", 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8860000000000001e-06 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635676+0000", 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1181999999999999e-05 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.635771+0000", 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5130000000000006e-05 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpRead(3.0s3 43/31 ECSubRead(tid=22, to_read={3:0353c118:::smithi12138290-10:head=0,40960,0}, subchunks={3:0353c118:::smithi12138290-10:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.573140+0000", 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.496418061, 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208385, 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.120 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.573140+0000", 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.573140+0000", 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.573141+0000", 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.573151+0000", 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0461e-05 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.573169+0000", 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7549000000000001e-05 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.573181+0000", 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2564e-05 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.573263+0000", 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2109999999999998e-05 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.573349+0000", 2024-09-16T03:40:26.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5439999999999995e-05 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 43/31 ECSubWrite(tid=28, reqid=client.4615.0:280, at_version=43'18, trim_to=0'0, roll_forward_to=42'16))", 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.575445+0000", 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.4941128589999999, 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014172340000000001, 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.124 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.575445+0000", 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.575445+0000", 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.575446+0000", 2024-09-16T03:40:26.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3e-07 2024-09-16T03:40:26.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.575545+0000", 2024-09-16T03:40:26.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9246000000000005e-05 2024-09-16T03:40:26.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.575557+0000", 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1969e-05 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.575565+0000", 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.354e-06 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.575618+0000", 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3168999999999999e-05 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.575625+0000", 2024-09-16T03:40:26.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2720000000000003e-06 2024-09-16T03:40:26.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.576805+0000", 2024-09-16T03:40:26.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011804960000000001 2024-09-16T03:40:26.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.576863+0000", 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7598e-05 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 43/31 ECSubWrite(tid=29, reqid=client.4615.0:280, at_version=0'0, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.577565+0000", 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.4919937619999999, 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000707031, 2024-09-16T03:40:26.129 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.577565+0000", 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.577565+0000", 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.577565+0000", 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.577576+0000", 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0626e-05 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.577588+0000", 2024-09-16T03:40:26.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.237e-05 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.577595+0000", 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0079999999999996e-06 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.577670+0000", 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4325000000000004e-05 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.577679+0000", 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0729999999999994e-06 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.578235+0000", 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055606600000000003 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.578272+0000", 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6838999999999997e-05 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.133 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 43/31 ECSubWrite(tid=34, reqid=client.4615.0:282, at_version=43'6, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.579052+0000", 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.490506119, 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002945193, 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579052+0000", 2024-09-16T03:40:26.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579052+0000", 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579052+0000", 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7499999999999999e-07 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579466+0000", 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000413664 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579484+0000", 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8238999999999999e-05 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579493+0000", 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7600000000000008e-06 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579570+0000", 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7218999999999994e-05 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579581+0000", 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0599e-05 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581952+0000", 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002371536 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581997+0000", 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5000999999999998e-05 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 43/31 ECSubWrite(tid=35, reqid=client.4615.0:283, at_version=43'7, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.581186+0000", 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.48837267, 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013244940000000001, 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.138 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.139 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581186+0000", 2024-09-16T03:40:26.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581186+0000", 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581186+0000", 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581458+0000", 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027135100000000002 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581471+0000", 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2765999999999999e-05 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581479+0000", 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2619999999999998e-06 2024-09-16T03:40:26.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581542+0000", 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3326999999999999e-05 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581556+0000", 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4049999999999999e-05 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.582480+0000", 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00092366799999999999 2024-09-16T03:40:26.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.582510+0000", 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0241000000000001e-05 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 43/31 ECSubWrite(tid=36, reqid=client.4615.0:284, at_version=43'8, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.583790+0000", 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.4857686120000002, 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00094956800000000003, 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.143 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.583790+0000", 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.583790+0000", 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.583791+0000", 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3300000000000001e-07 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.583956+0000", 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016564699999999999 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.583968+0000", 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.198e-05 2024-09-16T03:40:26.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.583975+0000", 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584029+0000", 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4015999999999999e-05 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584037+0000", 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6079999999999998e-06 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584709+0000", 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00067221399999999997 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584739+0000", 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0467000000000002e-05 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.147 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 43/31 ECSubWrite(tid=37, reqid=client.4615.0:285, at_version=43'9, trim_to=0'0, roll_forward_to=43'7))", 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.584917+0000", 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.4846415259999999, 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064698700000000002, 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584917+0000", 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584917+0000", 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584917+0000", 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4300000000000003e-07 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584931+0000", 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3672999999999999e-05 2024-09-16T03:40:26.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584942+0000", 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0927000000000001e-05 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584949+0000", 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5610000000000004e-06 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584992+0000", 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3871000000000001e-05 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.584999+0000", 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.195e-06 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.585535+0000", 2024-09-16T03:40:26.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00053676500000000003 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.585564+0000", 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8351999999999999e-05 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 43/31 ECSubWrite(tid=38, reqid=client.4615.0:285, at_version=0'0, trim_to=0'0, roll_forward_to=43'9))", 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.587060+0000", 2024-09-16T03:40:26.152 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.4824981510000002, 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065937700000000005, 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587060+0000", 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587060+0000", 2024-09-16T03:40:26.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587061+0000", 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587072+0000", 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1231e-05 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587082+0000", 2024-09-16T03:40:26.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9399999999999997e-06 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587088+0000", 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8470000000000001e-06 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587143+0000", 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4962e-05 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587150+0000", 2024-09-16T03:40:26.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-09-16T03:40:26.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587688+0000", 2024-09-16T03:40:26.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00053813999999999999 2024-09-16T03:40:26.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587720+0000", 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1093999999999997e-05 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpRead(3.4s3 43/31 ECSubRead(tid=39, to_read={3:2307c0f5:::smithi12138290-28:head=0,4096,0}, subchunks={3:2307c0f5:::smithi12138290-28:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.587514+0000", 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.4820441029999998, 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017170600000000001, 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.157 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587514+0000", 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587514+0000", 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587515+0000", 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587526+0000", 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1212999999999999e-05 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587540+0000", 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4184e-05 2024-09-16T03:40:26.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587552+0000", 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1951e-05 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587620+0000", 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7815e-05 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.587686+0000", 2024-09-16T03:40:26.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5951000000000006e-05 2024-09-16T03:40:26.161 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.161 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.161 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.161 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.161 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.161 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:26.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:40:26.163 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:26.160+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:26.221 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:26.263 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:26.261+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:26.307 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:26.307 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:26.307 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:26.307 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:26.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:40:26.364 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:26.363+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:26.458 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:26.458 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:26.458 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:26.458 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:26.459 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:26.465 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:26.463+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:26.466 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T03:40:26.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:40:26.471 INFO:tasks.rados.rados.0.smithi121.stdout:113: done (1 left) 2024-09-16T03:40:26.472 INFO:tasks.rados.rados.0.smithi121.stdout:114: done (0 left) 2024-09-16T03:40:26.472 INFO:tasks.rados.rados.0.smithi121.stdout:115: read oid 34 snap -1 2024-09-16T03:40:26.472 INFO:tasks.rados.rados.0.smithi121.stdout:115: expect (ObjNum 66 snap 3 seq_num 66) 2024-09-16T03:40:26.472 INFO:tasks.rados.rados.0.smithi121.stdout:116: snap_create 2024-09-16T03:40:26.485 INFO:tasks.rados.rados.0.smithi121.stdout:115: expect (ObjNum 66 snap 3 seq_num 66) 2024-09-16T03:40:26.566 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:26.564+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:26.624 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 37/31 ECSubWrite(tid=23, reqid=client.4615.0:205, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.045331040000001, 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050337250000000002, 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9999999999999999e-07 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526147+0000", 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00092697700000000005 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526170+0000", 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3377e-05 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526182+0000", 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1331999999999999e-05 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526338+0000", 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000156009 2024-09-16T03:40:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526347+0000", 2024-09-16T03:40:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7610000000000002e-06 2024-09-16T03:40:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530193+0000", 2024-09-16T03:40:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003845209 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530254+0000", 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0859999999999997e-05 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 37/31 ECSubWrite(tid=24, reqid=client.4615.0:206, at_version=37'15, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.527370+0000", 2024-09-16T03:40:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.043180822, 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003239432, 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527370+0000", 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527370+0000", 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527371+0000", 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1349999999999999e-06 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527479+0000", 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010815999999999999 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527788+0000", 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030877200000000002 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527802+0000", 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3419e-05 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527992+0000", 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019065999999999999 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528001+0000", 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0880000000000001e-06 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530572+0000", 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025707730000000002 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530609+0000", 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7425000000000002e-05 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 37/31 ECSubWrite(tid=25, reqid=client.4615.0:207, at_version=37'16, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.528569+0000", 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.041982131999999, 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025705290000000002, 2024-09-16T03:40:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528569+0000", 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528569+0000", 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528570+0000", 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9099999999999991e-07 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528591+0000", 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1586999999999998e-05 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528605+0000", 2024-09-16T03:40:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3759e-05 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528616+0000", 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.097e-05 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530007+0000", 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013907959999999999 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530014+0000", 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3069999999999997e-06 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.531107+0000", 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001092478 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.531139+0000", 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2641000000000001e-05 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=62, reqid=client.4615.0:225, at_version=38'9, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:17.542581+0000", 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.0279701919999997, 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031397600000000001, 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.542581+0000", 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.542581+0000", 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.542581+0000", 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4199999999999996e-07 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.543125+0000", 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054406500000000004 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.543139+0000", 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3536000000000001e-05 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.543151+0000", 2024-09-16T03:40:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2601e-05 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.543244+0000", 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2449999999999997e-05 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.543258+0000", 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4351e-05 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.545680+0000", 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002421856 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.545720+0000", 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0358999999999998e-05 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=63, reqid=client.4615.0:226, at_version=38'10, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:17.544005+0000", 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.0265460599999994, 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030870229999999999, 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544005+0000", 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544005+0000", 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544006+0000", 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4399999999999999e-07 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544532+0000", 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052680100000000003 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544547+0000", 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4226000000000001e-05 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544558+0000", 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0963e-05 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.545046+0000", 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048845799999999999 2024-09-16T03:40:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.545054+0000", 2024-09-16T03:40:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4060000000000005e-06 2024-09-16T03:40:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.547042+0000", 2024-09-16T03:40:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001987417 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.547092+0000", 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9907999999999997e-05 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/31 ECSubWrite(tid=64, reqid=client.4615.0:227, at_version=38'11, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:17.544771+0000", 2024-09-16T03:40:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.0257794249999996, 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026261460000000002, 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544771+0000", 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544771+0000", 2024-09-16T03:40:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544772+0000", 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.023e-06 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544792+0000", 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9259999999999999e-05 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544804+0000", 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1866e-05 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.544812+0000", 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6559999999999998e-06 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.545503+0000", 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069052299999999996 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.545515+0000", 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1868999999999999e-05 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.547361+0000", 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018462999999999999 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.547398+0000", 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6649000000000001e-05 2024-09-16T03:40:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.7s2 38/31 ECSubWrite(tid=10, reqid=client.4615.0:231, at_version=38'6, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:17.552101+0000", 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.0184496129999996, 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017107839999999999, 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.552101+0000", 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.552101+0000", 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.552101+0000", 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.36e-07 2024-09-16T03:40:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.552642+0000", 2024-09-16T03:40:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054058700000000003 2024-09-16T03:40:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.552655+0000", 2024-09-16T03:40:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2826999999999999e-05 2024-09-16T03:40:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.552663+0000", 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2099999999999993e-06 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.552716+0000", 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3242999999999998e-05 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.552723+0000", 2024-09-16T03:40:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.973e-06 2024-09-16T03:40:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.553775+0000", 2024-09-16T03:40:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00105163 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.553812+0000", 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7178000000000002e-05 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 38/31 ECSubWrite(tid=20, reqid=client.4615.0:229, at_version=38'11, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:17.553585+0000", 2024-09-16T03:40:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.0169659370000002, 2024-09-16T03:40:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003267931, 2024-09-16T03:40:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.553585+0000", 2024-09-16T03:40:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.553585+0000", 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.553585+0000", 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9500000000000001e-07 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554793+0000", 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001208375 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554811+0000", 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7518000000000001e-05 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554820+0000", 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0329999999999997e-06 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554896+0000", 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5914999999999994e-05 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554905+0000", 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5249999999999994e-06 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.556816+0000", 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019101669999999999 2024-09-16T03:40:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.556853+0000", 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7203000000000003e-05 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.7s2 38/31 ECSubWrite(tid=11, reqid=client.4615.0:232, at_version=38'7, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:17.554007+0000", 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.0165437589999993, 2024-09-16T03:40:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015965350000000001, 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554007+0000", 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554007+0000", 2024-09-16T03:40:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554008+0000", 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9699999999999996e-07 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554242+0000", 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023465700000000001 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554266+0000", 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3563000000000001e-05 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554305+0000", 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9180000000000001e-05 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554397+0000", 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1557000000000004e-05 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.554406+0000", 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2930000000000006e-06 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.555553+0000", 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001146916 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:17.555604+0000", 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0772000000000001e-05 2024-09-16T03:40:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 42/31 ECSubWrite(tid=22, reqid=osd.8.0:22, at_version=42'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.552536+0000", 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0180151520000003, 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018691580000000001, 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.552536+0000", 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.552536+0000", 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.552536+0000", 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2000000000000001e-07 2024-09-16T03:40:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.552565+0000", 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8614000000000001e-05 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.552577+0000", 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1983e-05 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.552588+0000", 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1515000000000001e-05 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.552665+0000", 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6842999999999996e-05 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.552676+0000", 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1372999999999999e-05 2024-09-16T03:40:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.554366+0000", 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016894970000000001 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.554405+0000", 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9013000000000001e-05 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 42/31 ECSubWrite(tid=23, reqid=osd.8.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.554994+0000", 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.015556428, 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020124320000000002, 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.554994+0000", 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.554994+0000", 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.554995+0000", 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8999999999999995e-07 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555007+0000", 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1599e-05 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555023+0000", 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6189e-05 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555033+0000", 2024-09-16T03:40:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4560000000000006e-06 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555102+0000", 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9434e-05 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555113+0000", 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1063999999999999e-05 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.556956+0000", 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018430650000000001 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557007+0000", 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0735000000000001e-05 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 42/31 ECSubWrite(tid=73, reqid=osd.0.0:73, at_version=42'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.555894+0000", 2024-09-16T03:40:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0146567519999996, 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041093190000000002, 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555894+0000", 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555894+0000", 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555894+0000", 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7100000000000001e-07 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555910+0000", 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5661000000000001e-05 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555921+0000", 2024-09-16T03:40:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1181e-05 2024-09-16T03:40:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555933+0000", 2024-09-16T03:40:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1949e-05 2024-09-16T03:40:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555984+0000", 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0584000000000002e-05 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555990+0000", 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5969999999999997e-06 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.559966+0000", 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039761299999999996 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.560003+0000", 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 42/31 ECSubWrite(tid=30, reqid=osd.14.0:30, at_version=42'20, trim_to=0'0, roll_forward_to=41'18))", 2024-09-16T03:40:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.557656+0000", 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0128949409999999, 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047645880000000002, 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557656+0000", 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557656+0000", 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557657+0000", 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.68e-07 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557674+0000", 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7887e-05 2024-09-16T03:40:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557696+0000", 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2035000000000002e-05 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557705+0000", 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5750000000000001e-06 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557774+0000", 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9015000000000002e-05 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557785+0000", 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1289e-05 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562388+0000", 2024-09-16T03:40:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004602783 2024-09-16T03:40:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562420+0000", 2024-09-16T03:40:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2335999999999997e-05 2024-09-16T03:40:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 42/31 ECSubWrite(tid=74, reqid=osd.0.0:73, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.562839+0000", 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0077113510000002, 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047921420000000001, 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562839+0000", 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562839+0000", 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562840+0000", 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9899999999999999e-07 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562852+0000", 2024-09-16T03:40:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1975e-05 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562863+0000", 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1100999999999999e-05 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562869+0000", 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9139999999999996e-06 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562921+0000", 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1564999999999998e-05 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562927+0000", 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.455e-06 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.567594+0000", 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046666099999999999 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.567632+0000", 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7722999999999998e-05 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 42/31 ECSubWrite(tid=32, reqid=osd.14.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.006825504, 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075866950000000001, 2024-09-16T03:40:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563726+0000", 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.09e-07 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563741+0000", 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5123000000000001e-05 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563761+0000", 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9335999999999999e-05 2024-09-16T03:40:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563770+0000", 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3770000000000002e-06 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563840+0000", 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9725000000000001e-05 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563851+0000", 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1107000000000001e-05 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571277+0000", 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074264830000000002 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571312+0000", 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4635000000000003e-05 2024-09-16T03:40:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/31 ECSubWrite(tid=23, reqid=client.4615.0:267, at_version=42'14, trim_to=0'0, roll_forward_to=34'11))", 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.628032+0000", 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.9425186559999998, 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037935640000000001, 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628032+0000", 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628032+0000", 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628033+0000", 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2800000000000005e-07 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628404+0000", 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037104 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628415+0000", 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0912e-05 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628421+0000", 2024-09-16T03:40:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628489+0000", 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8149000000000001e-05 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.628500+0000", 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1216999999999999e-05 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631794+0000", 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032933120000000001 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631826+0000", 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2146e-05 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/31 ECSubWrite(tid=24, reqid=client.4615.0:268, at_version=42'15, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.629148+0000", 2024-09-16T03:40:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.9414025109999997, 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053360220000000002, 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629148+0000", 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629148+0000", 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629149+0000", 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8400000000000003e-07 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629190+0000", 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0806999999999997e-05 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629227+0000", 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6551000000000003e-05 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629236+0000", 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7769999999999997e-06 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629292+0000", 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5980999999999998e-05 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629300+0000", 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4909999999999999e-06 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634449+0000", 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051495170000000002 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634484+0000", 2024-09-16T03:40:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5014000000000001e-05 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/31 ECSubWrite(tid=25, reqid=client.4615.0:269, at_version=42'16, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.9405541209999999, 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004694803, 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4900000000000005e-07 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630013+0000", 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5722e-05 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630025+0000", 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2234e-05 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630033+0000", 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3560000000000004e-06 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631638+0000", 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00160494 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631647+0000", 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4960000000000002e-06 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634669+0000", 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003022247 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634692+0000", 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2059e-05 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 43/31 ECSubWrite(tid=77, reqid=client.4615.0:275, at_version=43'15, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.568106+0000", 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0024451919999997, 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025397250000000001, 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.568106+0000", 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.568106+0000", 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.568106+0000", 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1899999999999999e-07 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.568531+0000", 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042484400000000001 2024-09-16T03:40:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.568549+0000", 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8771999999999999e-05 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.568561+0000", 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1847e-05 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.568644+0000", 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2968999999999998e-05 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.568657+0000", 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2576e-05 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.570610+0000", 2024-09-16T03:40:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019526680000000001 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.570645+0000", 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5830000000000001e-05 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 43/31 ECSubWrite(tid=78, reqid=client.4615.0:276, at_version=43'16, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.570282+0000", 2024-09-16T03:40:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0002693010000003, 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019279570000000001, 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.570282+0000", 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.570282+0000", 2024-09-16T03:40:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.570283+0000", 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0380000000000001e-06 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.570980+0000", 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069715700000000001 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.570992+0000", 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2357999999999999e-05 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.570999+0000", 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8970000000000002e-06 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.571054+0000", 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5470000000000003e-05 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.571066+0000", 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1861999999999999e-05 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.572166+0000", 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010998679999999999 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.572210+0000", 2024-09-16T03:40:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3306999999999997e-05 2024-09-16T03:40:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:26.721 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:26.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:40:26.723 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:26.720+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:26.823 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:26.821+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:26.827 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:26.827 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:26.827 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:26.827 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:26.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:40:26.925 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:26.924+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:26.980 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:26.980 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:26.980 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:26.980 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:26.980 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:26.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:40:27.026 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:27.025+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:27.094 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:27.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4341.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.830298804999998, 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59293511399999999, 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4341", 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-09-16T03:40:27.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260707+0000", 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.66e-06 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260713+0000", 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6150000000000002e-06 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260728+0000", 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5333e-05 2024-09-16T03:40:27.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260741+0000", 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260825+0000", 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4147999999999995e-05 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260836+0000", 2024-09-16T03:40:27.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0351e-05 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852485+0000", 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59164876099999997 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852573+0000", 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8352999999999999e-05 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852639+0000", 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853142+0000", 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050253099999999996 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853585+0000", 2024-09-16T03:40:27.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000443666 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853615+0000", 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9774e-05 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853640+0000", 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4924999999999999e-05 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717039+0000", 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.373964354999998, 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085512269999999998, 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-09-16T03:40:27.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717039+0000", 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717039+0000", 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717040+0000", 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2099999999999996e-07 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717044+0000", 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9299999999999996e-06 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717052+0000", 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717058+0000", 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3269999999999997e-06 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721385+0000", 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043270959999999999 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721439+0000", 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3418999999999998e-05 2024-09-16T03:40:27.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721511+0000", 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2670999999999996e-05 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725365+0000", 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038537179999999999 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725524+0000", 2024-09-16T03:40:27.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015862600000000001 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725566+0000", 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2425000000000001e-05 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725591+0000", 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4372e-05 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.107 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717067+0000", 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.373936458999999, 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086214770000000007, 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-09-16T03:40:27.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717067+0000", 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717067+0000", 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717068+0000", 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.75e-07 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717071+0000", 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5379999999999998e-06 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717077+0000", 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2430000000000002e-06 2024-09-16T03:40:27.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717082+0000", 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5879999999999999e-06 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721746+0000", 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046643919999999998 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721841+0000", 2024-09-16T03:40:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4192999999999998e-05 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721969+0000", 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012844500000000001 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725599+0000", 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036301150000000002 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725638+0000", 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9110000000000003e-05 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725665+0000", 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725689+0000", 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3546000000000001e-05 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.373889987999998, 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093665129999999999, 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.06e-07 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717170+0000", 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5937000000000003e-05 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717182+0000", 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1844999999999999e-05 2024-09-16T03:40:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717188+0000", 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.054e-06 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.724763+0000", 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075747119999999999 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.724809+0000", 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5933999999999998e-05 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.724882+0000", 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3650000000000001e-05 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726043+0000", 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011606260000000001 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726385+0000", 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034193800000000002 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726443+0000", 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8251999999999997e-05 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726480+0000", 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6959000000000003e-05 2024-09-16T03:40:27.119 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.120 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.120 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.120 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.120 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:27.120 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.373800747000001, 2024-09-16T03:40:27.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095750269999999998, 2024-09-16T03:40:27.120 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.120 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7800000000000001e-07 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717208+0000", 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.109e-06 2024-09-16T03:40:27.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717215+0000", 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0210000000000002e-06 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717226+0000", 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0934000000000001e-05 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725719+0000", 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084932659999999993 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725774+0000", 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5260000000000003e-05 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725850+0000", 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5907000000000004e-05 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726495+0000", 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064497100000000004 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726716+0000", 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022091799999999999 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726744+0000", 2024-09-16T03:40:27.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8561e-05 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726778+0000", 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3501999999999999e-05 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717235+0000", 2024-09-16T03:40:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.373768549000001, 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097518989999999996, 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717235+0000", 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717235+0000", 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717236+0000", 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8400000000000005e-07 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717240+0000", 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717247+0000", 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4939999999999997e-06 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717252+0000", 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726072+0000", 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088197359999999999 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726119+0000", 2024-09-16T03:40:27.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6662000000000001e-05 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726196+0000", 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6958000000000004e-05 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726593+0000", 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000397272 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726935+0000", 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034162800000000002 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726964+0000", 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9268000000000002e-05 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726987+0000", 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3096e-05 2024-09-16T03:40:27.132 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.939734805000001, 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094025229999999994, 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 162 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151271+0000", 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2510000000000001e-06 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151275+0000", 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4199999999999999e-06 2024-09-16T03:40:27.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151278+0000", 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8669999999999998e-06 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.156802+0000", 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055245119999999997 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.156852+0000", 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0195999999999998e-05 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.156927+0000", 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4192999999999999e-05 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159094+0000", 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021670740000000002 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160511+0000", 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001417583 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160599+0000", 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7965000000000002e-05 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160671+0000", 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2113999999999994e-05 2024-09-16T03:40:27.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.939721937000002, 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094948329999999994, 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 163 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.11e-07 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151284+0000", 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2680000000000001e-06 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151287+0000", 2024-09-16T03:40:27.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2820000000000001e-06 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151291+0000", 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2100000000000002e-06 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157351+0000", 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060598270000000003 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157437+0000", 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6279999999999994e-05 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157555+0000", 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118377 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159098+0000", 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015432499999999999 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160709+0000", 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016109799999999999 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160743+0000", 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3432000000000001e-05 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160777+0000", 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3716e-05 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.144 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151306+0000", 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.939697216999999, 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01219754, 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151306+0000", 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151306+0000", 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151307+0000", 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4999999999999999e-07 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151364+0000", 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7055000000000002e-05 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151371+0000", 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0550000000000002e-06 2024-09-16T03:40:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151374+0000", 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.534e-06 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157903+0000", 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065283620000000002 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157969+0000", 2024-09-16T03:40:27.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6211999999999996e-05 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158097+0000", 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127737 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159104+0000", 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010076919999999999 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163354+0000", 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042495579999999996 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163439+0000", 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5276000000000001e-05 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163504+0000", 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4808999999999996e-05 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.939622878000002, 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012275085, 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.151 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 165 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3799999999999998e-07 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151384+0000", 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5639999999999999e-06 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151388+0000", 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9280000000000003e-06 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151391+0000", 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0699999999999998e-06 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158639+0000", 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072479620000000002 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158736+0000", 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7435000000000005e-05 2024-09-16T03:40:27.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158872+0000", 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000136119 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159390+0000", 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051745999999999997 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163597+0000", 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042074870000000002 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163631+0000", 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4057000000000003e-05 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163656+0000", 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4664999999999998e-05 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.156 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:180 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.282924+0000", 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.808079881000001, 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092163079999999994, 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.157 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 180 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282924+0000", 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282924+0000", 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282924+0000", 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9099999999999999e-07 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282929+0000", 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3449999999999999e-06 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282933+0000", 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6720000000000003e-06 2024-09-16T03:40:27.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282936+0000", 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7580000000000001e-06 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.284410+0000", 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001474265 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.284472+0000", 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1357000000000002e-05 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.284588+0000", 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011673 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286974+0000", 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023862110000000001 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292099+0000", 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051242029999999999 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292123+0000", 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.402e-05 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292140+0000", 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7356000000000001e-05 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.162 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:181 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283040+0000", 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.807963616999999, 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092435739999999992, 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 181 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283040+0000", 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283040+0000", 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283041+0000", 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5499999999999998e-07 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283044+0000", 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8850000000000001e-06 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283052+0000", 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6240000000000002e-06 2024-09-16T03:40:27.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283057+0000", 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.481e-06 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.284980+0000", 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019231160000000001 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.285058+0000", 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8047999999999995e-05 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.285136+0000", 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8516000000000005e-05 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286980+0000", 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018439769999999999 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292190+0000", 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052099920000000001 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292243+0000", 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2831000000000003e-05 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292284+0000", 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0549000000000001e-05 2024-09-16T03:40:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:182 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283109+0000", 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.807895071000001, 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092439459999999994, 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 182 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283109+0000", 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283109+0000", 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283109+0000", 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7300000000000002e-07 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283183+0000", 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3855000000000004e-05 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283188+0000", 2024-09-16T03:40:27.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1510000000000003e-06 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283191+0000", 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.285359+0000", 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021678969999999998 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.285433+0000", 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3707000000000004e-05 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.285550+0000", 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116686 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288022+0000", 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024726729999999999 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292310+0000", 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042879900000000002 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292335+0000", 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4355999999999999e-05 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292353+0000", 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7983999999999999e-05 2024-09-16T03:40:27.174 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:183 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283197+0000", 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.807806288999998, 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093764810000000007, 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.175 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 183 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283197+0000", 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283197+0000", 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283198+0000", 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1199999999999999e-07 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283200+0000", 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1780000000000002e-06 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283203+0000", 2024-09-16T03:40:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.41e-06 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283206+0000", 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5730000000000002e-06 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286007+0000", 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028007969999999998 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286078+0000", 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1606000000000005e-05 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286188+0000", 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000109978 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288028+0000", 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018400619999999999 2024-09-16T03:40:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292533+0000", 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045046419999999997 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292556+0000", 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2733000000000001e-05 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292574+0000", 2024-09-16T03:40:27.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.819e-05 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:184 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.807794464000001, 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094925180000000001, 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.181 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 184 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0699999999999999e-07 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283211+0000", 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8810000000000001e-06 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283214+0000", 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1700000000000001e-06 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283219+0000", 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1989999999999999e-06 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286487+0000", 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032681239999999999 2024-09-16T03:40:27.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286562+0000", 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5328999999999997e-05 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286677+0000", 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114856 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289031+0000", 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023536310000000001 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292619+0000", 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035884580000000001 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292667+0000", 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8282000000000003e-05 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292702+0000", 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4381000000000002e-05 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.807775639999999, 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095539089999999993, 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.187 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 185 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4600000000000001e-07 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283260+0000", 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1488000000000001e-05 2024-09-16T03:40:27.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283264+0000", 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283267+0000", 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0299999999999998e-06 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287022+0000", 2024-09-16T03:40:27.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037554400000000001 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287090+0000", 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7329999999999994e-05 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287210+0000", 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119838 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289036+0000", 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001826814 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292735+0000", 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036988400000000001 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292762+0000", 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7036e-05 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292782+0000", 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9690999999999998e-05 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283295+0000", 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.807708487999999, 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0096111779999999997, 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.193 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 186 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283295+0000", 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283295+0000", 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283296+0000", 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0299999999999999e-07 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283299+0000", 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6500000000000002e-06 2024-09-16T03:40:27.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283304+0000", 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8790000000000002e-06 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283307+0000", 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.061e-06 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287556+0000", 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042482650000000002 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287625+0000", 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8989e-05 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287736+0000", 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111983 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289043+0000", 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013063020000000001 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292823+0000", 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037805619999999999 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292871+0000", 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7482999999999999e-05 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292906+0000", 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5500999999999998e-05 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.807691034000001, 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0096710279999999999, 2024-09-16T03:40:27.199 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283315+0000", 2024-09-16T03:40:27.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.227e-06 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283322+0000", 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2119999999999997e-06 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283325+0000", 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9000000000000002e-06 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288058+0000", 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047327790000000003 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288104+0000", 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5788999999999998e-05 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288174+0000", 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9769000000000002e-05 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289048+0000", 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087418399999999996 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292940+0000", 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038922129999999998 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292965+0000", 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4929000000000001e-05 2024-09-16T03:40:27.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292984+0000", 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8695999999999999e-05 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283335+0000", 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.807668315000001, 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097754880000000006, 2024-09-16T03:40:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 188 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283335+0000", 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283335+0000", 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283336+0000", 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8599999999999999e-07 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283372+0000", 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6279000000000003e-05 2024-09-16T03:40:27.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283376+0000", 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.454e-06 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283379+0000", 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288401+0000", 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050215930000000004 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288476+0000", 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4667999999999997e-05 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288596+0000", 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120439 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289367+0000", 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00077095699999999996 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293025+0000", 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036582390000000002 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293075+0000", 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0050999999999999e-05 2024-09-16T03:40:27.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293111+0000", 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5525999999999998e-05 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283384+0000", 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.807619385999999, 2024-09-16T03:40:27.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097953789999999999, 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 189 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283384+0000", 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283384+0000", 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283385+0000", 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7799999999999997e-07 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283387+0000", 2024-09-16T03:40:27.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2330000000000001e-06 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283391+0000", 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.878e-06 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283393+0000", 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.525e-06 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289066+0000", 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056729689999999999 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289107+0000", 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0513999999999999e-05 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289176+0000", 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9481999999999996e-05 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289693+0000", 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051673699999999997 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293139+0000", 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034465229999999999 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293163+0000", 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3811999999999998e-05 2024-09-16T03:40:27.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293180+0000", 2024-09-16T03:40:27.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6427999999999999e-05 2024-09-16T03:40:27.217 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.217 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.217 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.217 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.217 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.217 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:27.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:40:27.219 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:27.217+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:27.319 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:27.317+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:27.323 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:27.323 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:27.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:40:27.421 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:27.419+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:27.471 INFO:tasks.rados.rados.0.smithi121.stdout:115: done (1 left) 2024-09-16T03:40:27.471 INFO:tasks.rados.rados.0.smithi121.stdout:116: done (0 left) 2024-09-16T03:40:27.471 INFO:tasks.rados.rados.0.smithi121.stdout:117: snap_remove snap 5 2024-09-16T03:40:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:27.477 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:27.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:40:27.521 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:27.520+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:27.566 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:40:27.566 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:40:27.566 INFO:tasks.thrashosds.thrasher:inject_pause on osd.14 2024-09-16T03:40:27.566 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T03:40:27.566 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T03:40:27.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok config set filestore_inject_stall 3 2024-09-16T03:40:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:27.589 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:27.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:27.589 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 37/31 ECSubWrite(tid=23, reqid=client.4615.0:205, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.525429+0000", 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.061316926, 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030628029999999998, 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525429+0000", 2024-09-16T03:40:27.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525429+0000", 2024-09-16T03:40:27.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525429+0000", 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9399999999999999e-07 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526103+0000", 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00067450400000000003 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526122+0000", 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8408999999999999e-05 2024-09-16T03:40:27.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526165+0000", 2024-09-16T03:40:27.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3028000000000002e-05 2024-09-16T03:40:27.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526247+0000", 2024-09-16T03:40:27.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1771e-05 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526260+0000", 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2994e-05 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528441+0000", 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021813470000000001 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528492+0000", 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0556000000000002e-05 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 37/31 ECSubWrite(tid=24, reqid=client.4615.0:206, at_version=37'15, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.527492+0000", 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.059253739000001, 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029325269999999999, 2024-09-16T03:40:27.595 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527492+0000", 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527492+0000", 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527493+0000", 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527625+0000", 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000132347 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527638+0000", 2024-09-16T03:40:27.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2313999999999999e-05 2024-09-16T03:40:27.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.527648+0000", 2024-09-16T03:40:27.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0190000000000001e-05 2024-09-16T03:40:27.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528045+0000", 2024-09-16T03:40:27.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039730700000000001 2024-09-16T03:40:27.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528054+0000", 2024-09-16T03:40:27.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.568e-06 2024-09-16T03:40:27.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530372+0000", 2024-09-16T03:40:27.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002318213 2024-09-16T03:40:27.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530424+0000", 2024-09-16T03:40:27.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2497999999999998e-05 2024-09-16T03:40:27.600 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.600 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 37/31 ECSubWrite(tid=25, reqid=client.4615.0:207, at_version=37'16, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.528584+0000", 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.058161460999999, 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018881880000000001, 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528584+0000", 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528584+0000", 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528585+0000", 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.23e-07 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528601+0000", 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6345999999999999e-05 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528612+0000", 2024-09-16T03:40:27.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0474e-05 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528619+0000", 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6750000000000002e-06 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528721+0000", 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101844 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.528731+0000", 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0135e-05 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530430+0000", 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001698548 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530472+0000", 2024-09-16T03:40:27.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2443e-05 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 37/31 ECSubWrite(tid=26, reqid=client.4615.0:213, at_version=37'12, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.554991+0000", 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.031754224, 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022024309999999999, 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.554991+0000", 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.554991+0000", 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.554992+0000", 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7599999999999999e-07 2024-09-16T03:40:27.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.555362+0000", 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036991999999999998 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.555378+0000", 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6827999999999999e-05 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.555400+0000", 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1661000000000001e-05 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.555474+0000", 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3819000000000006e-05 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.555485+0000", 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1090000000000001e-05 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557105+0000", 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016205049999999999 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557194+0000", 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8431999999999996e-05 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.611 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 37/31 ECSubWrite(tid=27, reqid=client.4615.0:214, at_version=37'13, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:27.611 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.557158+0000", 2024-09-16T03:40:27.611 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.029587281, 2024-09-16T03:40:27.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030454010000000001, 2024-09-16T03:40:27.611 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.611 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.611 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.611 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557158+0000", 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557158+0000", 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557159+0000", 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2e-07 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557254+0000", 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5424e-05 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557265+0000", 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1225e-05 2024-09-16T03:40:27.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557274+0000", 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557338+0000", 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4554999999999994e-05 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557349+0000", 2024-09-16T03:40:27.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0986e-05 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.560166+0000", 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028167159999999999 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.560204+0000", 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7947000000000001e-05 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=27, reqid=client.4615.0:219, at_version=37'9, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.557507+0000", 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.029238756, 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027140049999999998, 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557507+0000", 2024-09-16T03:40:27.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557507+0000", 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557508+0000", 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8199999999999999e-07 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557523+0000", 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5146e-05 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557535+0000", 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2074e-05 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557545+0000", 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0623e-05 2024-09-16T03:40:27.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557602+0000", 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6864000000000003e-05 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.557613+0000", 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0257e-05 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.560184+0000", 2024-09-16T03:40:27.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025711940000000002 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.560221+0000", 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7165000000000001e-05 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 37/31 ECSubWrite(tid=28, reqid=client.4615.0:215, at_version=37'14, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.558035+0000", 2024-09-16T03:40:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.028710274, 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029844680000000001, 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.558035+0000", 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.558035+0000", 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.558036+0000", 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7899999999999998e-07 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.558051+0000", 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5021000000000001e-05 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.558064+0000", 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2663999999999999e-05 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.558072+0000", 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7829999999999999e-06 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.558130+0000", 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8372000000000003e-05 2024-09-16T03:40:27.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.558140+0000", 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8290000000000003e-06 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.560988+0000", 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028480089999999999 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.561020+0000", 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2110999999999998e-05 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=28, reqid=client.4615.0:219, at_version=0'0, trim_to=0'0, roll_forward_to=37'9))", 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.561482+0000", 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.025263562999999, 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001318905, 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.625 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.561482+0000", 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.561482+0000", 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.561483+0000", 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9199999999999999e-07 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.561496+0000", 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3033999999999999e-05 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.561508+0000", 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1795e-05 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.561517+0000", 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1970000000000002e-06 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.561578+0000", 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1027999999999999e-05 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.561587+0000", 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3540000000000006e-06 2024-09-16T03:40:27.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.562772+0000", 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001184755 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.562801+0000", 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8750000000000001e-05 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:220 3.3s0 3:c10296d1:::smithi12138290-49:head [delete] snapc 3=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.570699+0000", 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.016047106, 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022317159999999999, 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 220 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.630 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.570699+0000", 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.570699+0000", 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.570700+0000", 2024-09-16T03:40:27.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6479999999999999e-06 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.570706+0000", 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.570719+0000", 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3361e-05 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.570725+0000", 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2140000000000001e-06 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.570786+0000", 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0139000000000001e-05 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.570853+0000", 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7565999999999997e-05 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.571446+0000", 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059244600000000005 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.572212+0000", 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00076608899999999996 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.572872+0000", 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066021899999999998 2024-09-16T03:40:27.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.572930+0000", 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8414e-05 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 42/31 ECSubWrite(tid=30, reqid=osd.14.0:30, at_version=42'20, trim_to=0'0, roll_forward_to=41'18))", 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.557739+0000", 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.0290064829999999, 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045429850000000003, 2024-09-16T03:40:27.635 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557739+0000", 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557739+0000", 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557739+0000", 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4399999999999999e-07 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557774+0000", 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4845999999999998e-05 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557786+0000", 2024-09-16T03:40:27.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1697e-05 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557796+0000", 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0237e-05 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557870+0000", 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3464e-05 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557879+0000", 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6690000000000001e-06 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562242+0000", 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043627680000000004 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562282+0000", 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0160000000000002e-05 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.8s3 42/31 ECSubWrite(tid=183, reqid=osd.11.0:182, at_version=0'0, trim_to=0'0, roll_forward_to=42'8))", 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.560149+0000", 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.0265969100000003, 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032266069999999998, 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.560149+0000", 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.560149+0000", 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.560150+0000", 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5099999999999998e-07 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.560172+0000", 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2135000000000001e-05 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.560185+0000", 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3331e-05 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.560193+0000", 2024-09-16T03:40:27.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8010000000000004e-06 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.560246+0000", 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3548999999999998e-05 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.560253+0000", 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0990000000000004e-06 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563337+0000", 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030845249999999999 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563375+0000", 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8216000000000001e-05 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.644 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 42/31 ECSubWrite(tid=32, reqid=osd.14.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.563730+0000", 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.0230155730000003, 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050799820000000002, 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563730+0000", 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563730+0000", 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563731+0000", 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1200000000000002e-07 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563741+0000", 2024-09-16T03:40:27.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0399e-05 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563761+0000", 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9894999999999998e-05 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563771+0000", 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0237e-05 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563841+0000", 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9499000000000007e-05 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563850+0000", 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2229999999999996e-06 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.568773+0000", 2024-09-16T03:40:27.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004922605 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.568810+0000", 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7412e-05 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:257 3.3s0 3:c98c73f3:::smithi12138290-40:head [read 0~671744,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.582091+0000", 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.0046548319999999, 2024-09-16T03:40:27.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038254500000000002, 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 257 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582091+0000", 2024-09-16T03:40:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582091+0000", 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582093+0000", 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0209999999999998e-06 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582103+0000", 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0349e-05 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582118+0000", 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4457000000000001e-05 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582125+0000", 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4340000000000003e-06 2024-09-16T03:40:27.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582197+0000", 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2420000000000001e-05 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582305+0000", 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107735 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.585916+0000", 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036110339999999999 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:258 3.3s0 3:c98c73f3:::smithi12138290-40:head [read 0~671744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.582148+0000", 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.0045979349999996, 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042057680000000004, 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:40:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 258 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582148+0000", 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582148+0000", 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582148+0000", 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1799999999999995e-07 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582153+0000", 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9760000000000003e-06 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582161+0000", 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0369999999999995e-06 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582168+0000", 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.178e-06 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583254+0000", 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001086776 2024-09-16T03:40:27.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583313+0000", 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8485e-05 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.586353+0000", 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030406980000000001 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:259 3.3s0 3:c98c73f3:::smithi12138290-40:head [read 0~671744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.582176+0000", 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.0045697740000001, 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046375380000000001, 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 259 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.659 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582176+0000", 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582176+0000", 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582176+0000", 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3700000000000001e-07 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582178+0000", 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2510000000000001e-06 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582183+0000", 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2339999999999996e-06 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.582186+0000", 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0869999999999998e-06 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583683+0000", 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014973219999999999 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583783+0000", 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010039099999999999 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.586813+0000", 2024-09-16T03:40:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030299160000000001 2024-09-16T03:40:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.663 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpRead(3.3s0 42/31 ECSubRead(tid=16, to_read={3:c98c73f3:::smithi12138290-40:head=0,167936,0}, subchunks={3:c98c73f3:::smithi12138290-40:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.583179+0000", 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.0035668480000002, 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016624020000000001, 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583179+0000", 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583179+0000", 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583179+0000", 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583179+0000", 2024-09-16T03:40:27.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583198+0000", 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9690999999999998e-05 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583218+0000", 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9644e-05 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.584096+0000", 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087767699999999999 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.584841+0000", 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074538999999999996 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpReadReply(3.3s0 42/31 ECSubReadReply(tid=16, attrs_read=0))", 2024-09-16T03:40:27.667 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.583362+0000", 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.0033838199999998, 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001540451, 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583362+0000", 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583362+0000", 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583363+0000", 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5799999999999998e-07 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583607+0000", 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024432000000000002 2024-09-16T03:40:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583617+0000", 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0397000000000001e-05 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583625+0000", 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1729999999999998e-06 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.584867+0000", 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001242148 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.584902+0000", 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5555000000000001e-05 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpRead(3.3s0 42/31 ECSubRead(tid=17, to_read={3:c98c73f3:::smithi12138290-40:head=0,167936,0}, subchunks={3:c98c73f3:::smithi12138290-40:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.583605+0000", 2024-09-16T03:40:27.671 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.0031402089999997, 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013726000000000001, 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583605+0000", 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583605+0000", 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583605+0000", 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583605+0000", 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583622+0000", 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6328e-05 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.583630+0000", 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.191e-06 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.584921+0000", 2024-09-16T03:40:27.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012914599999999999 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.584978+0000", 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6620999999999999e-05 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s4 43/31 ECSubWrite(tid=34, reqid=client.4615.0:282, at_version=43'6, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.579129+0000", 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0076161859999999, 2024-09-16T03:40:27.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029488589999999999, 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579129+0000", 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579129+0000", 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579130+0000", 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7399999999999999e-07 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579466+0000", 2024-09-16T03:40:27.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000336332 2024-09-16T03:40:27.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579482+0000", 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5773999999999998e-05 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579498+0000", 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6135999999999999e-05 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579575+0000", 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7353000000000003e-05 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579585+0000", 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7180000000000008e-06 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.582034+0000", 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002449176 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.582078+0000", 2024-09-16T03:40:27.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4095999999999999e-05 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s4 43/31 ECSubWrite(tid=35, reqid=client.4615.0:283, at_version=43'7, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.581093+0000", 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0056529459999997, 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013347630000000001, 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:27.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581093+0000", 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581093+0000", 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:27.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581097+0000", 2024-09-16T03:40:27.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2250000000000002e-06 2024-09-16T03:40:27.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:27.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581344+0000", 2024-09-16T03:40:27.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024677199999999997 2024-09-16T03:40:27.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581357+0000", 2024-09-16T03:40:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2833999999999999e-05 2024-09-16T03:40:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581366+0000", 2024-09-16T03:40:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1439999999999992e-06 2024-09-16T03:40:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581425+0000", 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9607e-05 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581434+0000", 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1220000000000001e-06 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.582367+0000", 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093278000000000002 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:27.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.582427+0000", 2024-09-16T03:40:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0278999999999997e-05 2024-09-16T03:40:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:27.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:27.687 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:27.688 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:27.688 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T03:40:27.688 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:27.689 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:27.686+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:27.689 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:40:27.789 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:27.787+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:27.802 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:27.802 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:27.802 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:27.802 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:27.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:40:27.895 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:27.894+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:27.957 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:27.957 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:27.965 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:40:27.990 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:27.989+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:28.091 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:28.090+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:28.110 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:202 3.cs0 3:335b675a:::smithi12138290-38:head [read 0~622592,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.517297+0000", 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.590700655999999, 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030985510000000002, 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.111 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 202 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517297+0000", 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517297+0000", 2024-09-16T03:40:28.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517299+0000", 2024-09-16T03:40:28.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.012e-06 2024-09-16T03:40:28.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517309+0000", 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4069999999999999e-06 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517323+0000", 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3757000000000001e-05 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517333+0000", 2024-09-16T03:40:28.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0183000000000001e-05 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517387+0000", 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4616e-05 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517393+0000", 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4600000000000002e-06 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.518733+0000", 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001339891 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.518789+0000", 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6307999999999997e-05 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520396+0000", 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606917 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:203 3.cs0 3:335b675a:::smithi12138290-38:head [read 0~622592,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.517358+0000", 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.590640130000001, 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035734669999999999, 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.117 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 203 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517358+0000", 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517358+0000", 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517358+0000", 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517363+0000", 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2089999999999999e-06 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517368+0000", 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.361e-06 2024-09-16T03:40:28.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517371+0000", 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4450000000000001e-06 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517449+0000", 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7540999999999995e-05 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:28.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517455+0000", 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519211+0000", 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017551070000000001 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.519297+0000", 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6359000000000004e-05 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520931+0000", 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016344980000000001 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:204 3.cs0 3:335b675a:::smithi12138290-38:head [read 0~622592,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.517379+0000", 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.590618987999999, 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047868290000000003, 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.122 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 204 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517379+0000", 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517379+0000", 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517379+0000", 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1199999999999999e-07 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517381+0000", 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8789999999999999e-06 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517385+0000", 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.687e-06 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517388+0000", 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9079999999999999e-06 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517480+0000", 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2122999999999999e-05 2024-09-16T03:40:28.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.517483+0000", 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.520973+0000", 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034891940000000001 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.521058+0000", 2024-09-16T03:40:28.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5296999999999993e-05 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.522166+0000", 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011079499999999999 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 37/31 ECSubWrite(tid=23, reqid=client.4615.0:205, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.524999+0000", 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.582998974000001, 2024-09-16T03:40:28.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014483740000000001, 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.524999+0000", 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.524999+0000", 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.524999+0000", 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4600000000000001e-07 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525283+0000", 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028382599999999999 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525294+0000", 2024-09-16T03:40:28.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0754000000000001e-05 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525299+0000", 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2800000000000003e-06 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525340+0000", 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0621000000000003e-05 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525348+0000", 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0169999999999996e-06 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526413+0000", 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00106529 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.526447+0000", 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4440000000000002e-05 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.131 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:209 3.cs0 3:3e925a6e:::smithi12138290-43:head [delete] snapc 3=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.525370+0000", 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.582628005, 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040223389999999998, 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.132 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 209 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525370+0000", 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525370+0000", 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525371+0000", 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525374+0000", 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4709999999999999e-06 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525381+0000", 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0090000000000001e-06 2024-09-16T03:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525386+0000", 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525429+0000", 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2889000000000002e-05 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525480+0000", 2024-09-16T03:40:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1023999999999998e-05 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525962+0000", 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00048260600000000001 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528746+0000", 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027839470000000002 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529350+0000", 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00060369600000000003 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529392+0000", 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2571000000000003e-05 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 37/31 ECSubWrite(tid=24, reqid=client.4615.0:206, at_version=37'15, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.527047+0000", 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.58095103, 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020053649999999998, 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527047+0000", 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527047+0000", 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527048+0000", 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7700000000000004e-07 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527281+0000", 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023318699999999999 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527292+0000", 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0698e-05 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527300+0000", 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2290000000000003e-06 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527350+0000", 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.956e-05 2024-09-16T03:40:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.527358+0000", 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4230000000000005e-06 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529016+0000", 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001657857 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.529052+0000", 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6634000000000002e-05 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:225 3.fs0 3:fe9db9b8:::smithi12138290-24:head [append 0~720896] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.538459+0000", 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.569539431999999, 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076796640000000001, 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.142 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 225 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.538459+0000", 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.538459+0000", 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.538462+0000", 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2990000000000001e-06 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539454+0000", 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00099185099999999997 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539477+0000", 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3056000000000001e-05 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539487+0000", 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0019e-05 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539564+0000", 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6762999999999997e-05 2024-09-16T03:40:28.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539570+0000", 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5359999999999998e-06 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541406+0000", 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001836086 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541468+0000", 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1659e-05 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542536+0000", 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001067597 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545013+0000", 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002477061 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.546056+0000", 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00104386 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.546138+0000", 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1877000000000002e-05 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.148 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:226 3.fs0 3:fe9db9b8:::smithi12138290-24:head [append 0~376832] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.567544, 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095498700000000002, 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 226 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540455+0000", 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:40:28.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540861+0000", 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040549800000000001 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540874+0000", 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2974e-05 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540882+0000", 2024-09-16T03:40:28.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540948+0000", 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6383999999999993e-05 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540954+0000", 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9730000000000002e-06 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543210+0000", 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022561209999999998 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543290+0000", 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9907999999999995e-05 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543922+0000", 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000631713 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545016+0000", 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001093896 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.549959+0000", 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049431980000000002 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550004+0000", 2024-09-16T03:40:28.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4858999999999999e-05 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:227 3.fs0 3:fe9db9b8:::smithi12138290-24:head [setxattr _header (56) in=7b] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.567090358, 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092805639999999998, 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.155 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 227 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:28.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540909+0000", 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6400000000000001e-07 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540912+0000", 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1439999999999999e-06 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540919+0000", 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4000000000000003e-06 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540925+0000", 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6069999999999998e-06 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541006+0000", 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1231999999999997e-05 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541012+0000", 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3609999999999997e-06 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544371+0000", 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033583739999999999 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544424+0000", 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3535000000000002e-05 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544709+0000", 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028452099999999998 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545291+0000", 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00058281499999999998 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550110+0000", 2024-09-16T03:40:28.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048181430000000004 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550188+0000", 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8668e-05 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:228 3.fs0 3:fe9db9b8:::smithi12138290-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.567049204, 2024-09-16T03:40:28.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011210417, 2024-09-16T03:40:28.162 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.162 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.162 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:28.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.162 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 228 2024-09-16T03:40:28.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.162 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0399999999999996e-07 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540952+0000", 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6860000000000002e-06 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540960+0000", 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9729999999999997e-06 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540966+0000", 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7509999999999997e-06 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541041+0000", 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5216999999999995e-05 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541046+0000", 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0000000000000004e-06 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544875+0000", 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038287339999999999 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544964+0000", 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9245000000000004e-05 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550540+0000", 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055763649999999998 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550618+0000", 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7908e-05 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552159+0000", 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015409340000000001 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.167 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 42/31 ECSubWrite(tid=30, reqid=osd.14.0:30, at_version=42'20, trim_to=0'0, roll_forward_to=41'18))", 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.557504+0000", 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.5504941389999995, 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023794039999999999, 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557504+0000", 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557504+0000", 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557504+0000", 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557524+0000", 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9799000000000001e-05 2024-09-16T03:40:28.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557540+0000", 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6252e-05 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557549+0000", 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2099999999999993e-06 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557604+0000", 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5668999999999998e-05 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557615+0000", 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0637e-05 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.559858+0000", 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022429720000000002 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.559883+0000", 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5522999999999999e-05 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.8s5 42/31 ECSubWrite(tid=183, reqid=osd.11.0:182, at_version=0'0, trim_to=0'0, roll_forward_to=42'8))", 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.560093+0000", 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.5479049920000003, 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002379389, 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.172 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.560093+0000", 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.560093+0000", 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.560094+0000", 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2099999999999996e-07 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.560102+0000", 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0949999999999996e-06 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.560113+0000", 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0916e-05 2024-09-16T03:40:28.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.560120+0000", 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5750000000000003e-06 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.560161+0000", 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1168999999999998e-05 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.560166+0000", 2024-09-16T03:40:28.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6330000000000004e-06 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.562430+0000", 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00226341 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.562472+0000", 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2969999999999997e-05 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.176 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:256 3.cs0 3:38c596e1:::smithi12138290-26:head [delete] snapc 6=[5,3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579083+0000", 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.5289150280000001, 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027515539999999998, 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.177 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 256 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579083+0000", 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579083+0000", 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579085+0000", 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.609e-06 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579089+0000", 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.391e-06 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579102+0000", 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2609000000000001e-05 2024-09-16T03:40:28.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579110+0000", 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9500000000000001e-06 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579175+0000", 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4949000000000004e-05 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579274+0000", 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9640000000000001e-05 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579832+0000", 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00055801400000000001 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.580701+0000", 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00086885100000000002 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581735+0000", 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010341580000000001 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581835+0000", 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9382999999999994e-05 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.182 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:275 3.fs0 3:fe9db9b8:::smithi12138290-24:head [append 0~442368] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:22.564756+0000", 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.543242233, 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071858640000000001, 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 275 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.564756+0000", 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.564756+0000", 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.564758+0000", 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6299999999999998e-06 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.565293+0000", 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00053470299999999998 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.565316+0000", 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2563000000000001e-05 2024-09-16T03:40:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.565329+0000", 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3168e-05 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.565407+0000", 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8417000000000005e-05 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.565417+0000", 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5810000000000003e-06 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.567131+0000", 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017139060000000001 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.567206+0000", 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5300000000000001e-05 2024-09-16T03:40:28.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.568011+0000", 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00080480000000000005 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569277+0000", 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001266128 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:28.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.571849+0000", 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025717610000000001 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.571942+0000", 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2906999999999997e-05 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:276 3.fs0 3:fe9db9b8:::smithi12138290-24:head [append 0~720896] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:28.189 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:22.568707+0000", 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.5392914910000002, 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044533309999999996, 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 276 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.568707+0000", 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.568707+0000", 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.568708+0000", 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7439999999999999e-06 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569378+0000", 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00066960299999999995 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569391+0000", 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2554e-05 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569400+0000", 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1689999999999998e-06 2024-09-16T03:40:28.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569451+0000", 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0844000000000003e-05 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569512+0000", 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0998000000000001e-05 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570149+0000", 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00063699299999999998 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.571665+0000", 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001516784 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.573100+0000", 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00143443 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.573160+0000", 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0211999999999999e-05 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:277 3.fs0 3:fe9db9b8:::smithi12138290-24:head [append 0~49152] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:22.569421+0000", 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.5385767299999999, 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040546549999999999, 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.195 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 277 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569421+0000", 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569421+0000", 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569422+0000", 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4099999999999997e-07 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569470+0000", 2024-09-16T03:40:28.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8238000000000001e-05 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569479+0000", 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3180000000000007e-06 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569485+0000", 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3450000000000002e-06 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570729+0000", 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012436509999999999 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570821+0000", 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1974999999999999e-05 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.571254+0000", 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000433137 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.572730+0000", 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001475954 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.573390+0000", 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00066029499999999998 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.573476+0000", 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5901000000000003e-05 2024-09-16T03:40:28.200 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:278 3.fs0 3:fe9db9b8:::smithi12138290-24:head [setxattr _header (56) in=7b] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:22.569498+0000", 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.5384996399999995, 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044274509999999998, 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.201 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 278 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569498+0000", 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569498+0000", 2024-09-16T03:40:28.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569499+0000", 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1600000000000001e-07 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569503+0000", 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2429999999999999e-06 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569511+0000", 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4000000000000003e-06 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569516+0000", 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5180000000000004e-06 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.571982+0000", 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024655530000000001 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.572036+0000", 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3684999999999998e-05 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.572331+0000", 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000295591 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.573017+0000", 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00068587499999999996 2024-09-16T03:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.573875+0000", 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000857502 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.573926+0000", 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1368e-05 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:279 3.fs0 3:fe9db9b8:::smithi12138290-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:22.569524+0000", 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.5384741709999998, 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056959649999999999, 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 279 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.207 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569524+0000", 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569524+0000", 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569524+0000", 2024-09-16T03:40:28.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7899999999999999e-07 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569532+0000", 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3599999999999998e-06 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569539+0000", 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2439999999999996e-06 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.569544+0000", 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4600000000000002e-06 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.572757+0000", 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032129189999999998 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.572838+0000", 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0599000000000005e-05 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.574197+0000", 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013585540000000001 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.574248+0000", 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1165999999999998e-05 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.575220+0000", 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097218399999999996 2024-09-16T03:40:28.211 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 43/31 ECSubWriteReply(tid=77, last_complete=43'15, committed=1, applied=1))", 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:22.570179+0000", 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.5378189720000002, 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001431917, 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570179+0000", 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570179+0000", 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570180+0000", 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0130000000000001e-06 2024-09-16T03:40:28.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570186+0000", 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9410000000000004e-06 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570199+0000", 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3305000000000001e-05 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570209+0000", 2024-09-16T03:40:28.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.571596+0000", 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013875580000000001 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.571611+0000", 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4628e-05 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.215 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 43/31 ECSubWriteReply(tid=77, last_complete=43'15, committed=1, applied=1))", 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:22.570221+0000", 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.5377773320000001, 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001447891, 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570221+0000", 2024-09-16T03:40:28.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570221+0000", 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570221+0000", 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7399999999999998e-07 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570225+0000", 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2250000000000002e-06 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570240+0000", 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4080000000000001e-05 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.570248+0000", 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0330000000000008e-06 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.571645+0000", 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001397548 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.571669+0000", 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3530999999999999e-05 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.219 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:28.221 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:40:28.221 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:28.219+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:28.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:28.321 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:28.319+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:28.327 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:28.328 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:28.337 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:40:28.422 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:28.420+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:28.443 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:28.443 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:28.450 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:40:28.478 INFO:tasks.rados.rados.0.smithi121.stdout:117: done (0 left) 2024-09-16T03:40:28.478 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 30 current snap is 11 2024-09-16T03:40:28.478 INFO:tasks.rados.rados.0.smithi121.stdout:118: seq_num 75 ranges {491520=491520,983040=770048} 2024-09-16T03:40:28.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:40:28.483 INFO:tasks.rados.rados.0.smithi121.stdout:118: writing smithi12138290-30 from 491520 to 983040 tid 1 2024-09-16T03:40:28.487 INFO:tasks.rados.rados.0.smithi121.stdout:118: writing smithi12138290-30 from 983040 to 1753088 tid 2 2024-09-16T03:40:28.487 INFO:tasks.rados.rados.0.smithi121.stdout:119: rollback oid 18 current snap is 11 2024-09-16T03:40:28.487 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 18 to 3 2024-09-16T03:40:28.488 INFO:tasks.rados.rados.0.smithi121.stdout:120: setattr oid 34 current snap is 11 2024-09-16T03:40:28.491 INFO:tasks.rados.rados.0.smithi121.stdout:121: copy_from oid 1 from oid 31 current snap is 11 2024-09-16T03:40:28.491 INFO:tasks.rados.rados.0.smithi121.stdout:122: rollback oid 3 current snap is 11 2024-09-16T03:40:28.492 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 3 to 8 2024-09-16T03:40:28.492 INFO:tasks.rados.rados.0.smithi121.stdout:123: delete oid 4 current snap is 11 2024-09-16T03:40:28.497 INFO:tasks.rados.rados.0.smithi121.stdout:119: finishing rollback tid 0 to smithi12138290-18 2024-09-16T03:40:28.497 INFO:tasks.rados.rados.0.smithi121.stdout:118: finishing write tid 1 to smithi12138290-30 2024-09-16T03:40:28.498 INFO:tasks.rados.rados.0.smithi121.stdout:119: finishing rollback tid 1 to smithi12138290-18 2024-09-16T03:40:28.498 INFO:tasks.rados.rados.0.smithi121.stdout:119: finishing rollback tid 2 to smithi12138290-18 2024-09-16T03:40:28.498 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 18 v 24 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:40:28.502 INFO:tasks.rados.rados.0.smithi121.stdout:119: done (5 left) 2024-09-16T03:40:28.502 INFO:tasks.rados.rados.0.smithi121.stdout:123: done (4 left) 2024-09-16T03:40:28.502 INFO:tasks.rados.rados.0.smithi121.stdout:124: snap_remove snap 7 2024-09-16T03:40:28.523 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:28.522+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:28.566 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:28.566 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:28.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 37/31 ECSubWrite(tid=26, reqid=client.4615.0:213, at_version=37'12, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.554718+0000", 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.008650397, 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014147280000000001, 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.554718+0000", 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.554718+0000", 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.554718+0000", 2024-09-16T03:40:28.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.46e-07 2024-09-16T03:40:28.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.569 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555147+0000", 2024-09-16T03:40:28.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00042841 2024-09-16T03:40:28.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555157+0000", 2024-09-16T03:40:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9189999999999994e-06 2024-09-16T03:40:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555180+0000", 2024-09-16T03:40:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3561999999999999e-05 2024-09-16T03:40:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555225+0000", 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5281999999999998e-05 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555233+0000", 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.576e-06 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556107+0000", 2024-09-16T03:40:28.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00087447499999999999 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556133+0000", 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5157999999999999e-05 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=25, reqid=client.4615.0:217, at_version=37'7, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.555382+0000", 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.007985872000001, 2024-09-16T03:40:28.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010018010000000001, 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555382+0000", 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555382+0000", 2024-09-16T03:40:28.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555383+0000", 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.02e-07 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555397+0000", 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4606999999999999e-05 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555406+0000", 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5080000000000005e-06 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555413+0000", 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.289e-06 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555477+0000", 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3931999999999997e-05 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.555486+0000", 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5179999999999993e-06 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556355+0000", 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00086894499999999998 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556384+0000", 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8900000000000001e-05 2024-09-16T03:40:28.576 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 37/31 ECSubWrite(tid=27, reqid=client.4615.0:214, at_version=37'13, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.556843+0000", 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.006524838000001, 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00078009200000000002, 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.577 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.578 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.578 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.578 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556843+0000", 2024-09-16T03:40:28.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556843+0000", 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556844+0000", 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556982+0000", 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138009 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556991+0000", 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.867e-06 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.556996+0000", 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.361e-06 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557033+0000", 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6655000000000001e-05 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557038+0000", 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8029999999999996e-06 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557601+0000", 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056317100000000001 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.582 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.582 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557624+0000", 2024-09-16T03:40:28.582 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2665999999999999e-05 2024-09-16T03:40:28.582 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.582 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.582 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.582 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=27, reqid=client.4615.0:219, at_version=37'9, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:28.582 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.557360+0000", 2024-09-16T03:40:28.583 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.006008438, 2024-09-16T03:40:28.583 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001050537, 2024-09-16T03:40:28.583 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.583 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.583 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.584 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.584 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.584 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557360+0000", 2024-09-16T03:40:28.584 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.585 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.585 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.585 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557360+0000", 2024-09-16T03:40:28.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.585 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.585 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.585 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557360+0000", 2024-09-16T03:40:28.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2799999999999996e-07 2024-09-16T03:40:28.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.586 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557372+0000", 2024-09-16T03:40:28.586 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1201e-05 2024-09-16T03:40:28.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557380+0000", 2024-09-16T03:40:28.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0369999999999995e-06 2024-09-16T03:40:28.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.587 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557386+0000", 2024-09-16T03:40:28.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1380000000000004e-06 2024-09-16T03:40:28.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557422+0000", 2024-09-16T03:40:28.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6093000000000002e-05 2024-09-16T03:40:28.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557428+0000", 2024-09-16T03:40:28.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9939999999999997e-06 2024-09-16T03:40:28.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.558381+0000", 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00095336599999999998 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.558410+0000", 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9179999999999998e-05 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.590 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 37/31 ECSubWrite(tid=28, reqid=client.4615.0:215, at_version=37'14, trim_to=0'0, roll_forward_to=34'10))", 2024-09-16T03:40:28.590 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.557971+0000", 2024-09-16T03:40:28.590 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.005397407, 2024-09-16T03:40:28.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013109319999999999, 2024-09-16T03:40:28.590 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.590 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.590 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.590 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.591 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.591 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557971+0000", 2024-09-16T03:40:28.591 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.591 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.591 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557971+0000", 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557971+0000", 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3799999999999997e-07 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557986+0000", 2024-09-16T03:40:28.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4572e-05 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557994+0000", 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.824e-06 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.557998+0000", 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5260000000000004e-06 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.558036+0000", 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7755e-05 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.558041+0000", 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2440000000000001e-06 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.559256+0000", 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012145929999999999 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.559282+0000", 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.588e-05 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=28, reqid=client.4615.0:219, at_version=0'0, trim_to=0'0, roll_forward_to=37'9))", 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.561461+0000", 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.001907151999999, 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012191909999999999, 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.595 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561461+0000", 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561461+0000", 2024-09-16T03:40:28.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561462+0000", 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561469+0000", 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6429999999999995e-06 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561479+0000", 2024-09-16T03:40:28.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6770000000000007e-06 2024-09-16T03:40:28.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561487+0000", 2024-09-16T03:40:28.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4270000000000002e-06 2024-09-16T03:40:28.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561538+0000", 2024-09-16T03:40:28.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1001000000000002e-05 2024-09-16T03:40:28.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.599 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.561543+0000", 2024-09-16T03:40:28.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7019999999999998e-06 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.562657+0000", 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011138770000000001 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.562680+0000", 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3235e-05 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 37/31 ECSubWrite(tid=14, reqid=client.4615.0:220, at_version=37'10, trim_to=0'0, roll_forward_to=34'8))", 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.571440+0000", 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.991928271000001, 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001255831, 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571440+0000", 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571440+0000", 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571440+0000", 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2499999999999999e-07 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571461+0000", 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.065e-05 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571469+0000", 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2560000000000002e-06 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571475+0000", 2024-09-16T03:40:28.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6729999999999997e-06 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571533+0000", 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8482e-05 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.571542+0000", 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2820000000000002e-06 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.572664+0000", 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001121562 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.572696+0000", 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1801000000000003e-05 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.605 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 37/31 ECSubWrite(tid=15, reqid=client.4615.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:28.606 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.573055+0000", 2024-09-16T03:40:28.606 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.990313009999999, 2024-09-16T03:40:28.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00069386000000000003, 2024-09-16T03:40:28.606 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.606 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.606 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.606 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573055+0000", 2024-09-16T03:40:28.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573055+0000", 2024-09-16T03:40:28.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573056+0000", 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7299999999999995e-07 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573066+0000", 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0499999999999999e-05 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573081+0000", 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4219999999999999e-05 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573089+0000", 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.225e-06 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573144+0000", 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5498999999999998e-05 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573154+0000", 2024-09-16T03:40:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0087e-05 2024-09-16T03:40:28.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573719+0000", 2024-09-16T03:40:28.610 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056409400000000001 2024-09-16T03:40:28.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.573749+0000", 2024-09-16T03:40:28.610 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0562000000000003e-05 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.es3 38/31 ECSubRead(tid=29, to_read={3:7acde04a:::smithi12138290-46:head=0,192512,0}, subchunks={3:7acde04a:::smithi12138290-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.545569+0000", 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.017799179000001, 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000662844, 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.611 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545569+0000", 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545569+0000", 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545569+0000", 2024-09-16T03:40:28.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5100000000000001e-07 2024-09-16T03:40:28.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545579+0000", 2024-09-16T03:40:28.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3600000000000002e-06 2024-09-16T03:40:28.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545594+0000", 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5387e-05 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545604+0000", 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0182e-05 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545683+0000", 2024-09-16T03:40:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9118999999999999e-05 2024-09-16T03:40:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.546232+0000", 2024-09-16T03:40:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000548545 2024-09-16T03:40:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.0s2 42/31 ECSubWrite(tid=20, reqid=client.4615.0:251, at_version=42'11, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.564339+0000", 2024-09-16T03:40:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.9990297640000003, 2024-09-16T03:40:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027709689999999999, 2024-09-16T03:40:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.564339+0000", 2024-09-16T03:40:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.564339+0000", 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.564339+0000", 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9700000000000002e-07 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.564439+0000", 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9932000000000005e-05 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.564449+0000", 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.006e-05 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.564458+0000", 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7600000000000008e-06 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.564515+0000", 2024-09-16T03:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7417999999999998e-05 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.564522+0000", 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6189999999999996e-06 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.567077+0000", 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002555444 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.567109+0000", 2024-09-16T03:40:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2339000000000003e-05 2024-09-16T03:40:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.0s2 42/31 ECSubWrite(tid=21, reqid=client.4615.0:251, at_version=0'0, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T03:40:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.570008+0000", 2024-09-16T03:40:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.9933604689999997, 2024-09-16T03:40:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057574600000000003, 2024-09-16T03:40:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.570008+0000", 2024-09-16T03:40:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.570008+0000", 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.570008+0000", 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.570020+0000", 2024-09-16T03:40:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1453e-05 2024-09-16T03:40:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.570030+0000", 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7070000000000004e-06 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.570036+0000", 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9320000000000001e-06 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.570083+0000", 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7469000000000002e-05 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.570092+0000", 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0659999999999993e-06 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.570554+0000", 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00046141900000000002 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.570584+0000", 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0023e-05 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.ds1 42/31 ECSubRead(tid=23, to_read={3:b952e66c:::smithi12138290-29:head=0,118784,0}, subchunks={3:b952e66c:::smithi12138290-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.582549+0000", 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.9808192829999998, 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062770199999999995, 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.628 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582549+0000", 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582549+0000", 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582549+0000", 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7099999999999998e-07 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582556+0000", 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3749999999999999e-06 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582569+0000", 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3196e-05 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582580+0000", 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1423000000000001e-05 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582668+0000", 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7979999999999995e-05 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583177+0000", 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00050845700000000003 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpRead(3.ds1 42/31 ECSubRead(tid=24, to_read={3:b952e66c:::smithi12138290-29:head=0,118784,0}, subchunks={3:b952e66c:::smithi12138290-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.582625+0000", 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.9807436129999996, 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067553800000000005, 2024-09-16T03:40:28.632 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.633 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.633 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.633 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.633 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.633 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582625+0000", 2024-09-16T03:40:28.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.633 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582625+0000", 2024-09-16T03:40:28.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582625+0000", 2024-09-16T03:40:28.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4900000000000001e-07 2024-09-16T03:40:28.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582631+0000", 2024-09-16T03:40:28.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1879999999999999e-06 2024-09-16T03:40:28.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582646+0000", 2024-09-16T03:40:28.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4569999999999999e-05 2024-09-16T03:40:28.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.582671+0000", 2024-09-16T03:40:28.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5108000000000001e-05 2024-09-16T03:40:28.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.637 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583211+0000", 2024-09-16T03:40:28.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054007600000000001 2024-09-16T03:40:28.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583300+0000", 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9246999999999994e-05 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.9s4 42/31 ECSubWrite(tid=31, reqid=client.4615.0:260, at_version=42'12, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.590463+0000", 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.9729052429999996, 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00176402, 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.638 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.590463+0000", 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.590463+0000", 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.590464+0000", 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1699999999999998e-07 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.590896+0000", 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043241900000000002 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.590909+0000", 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3331e-05 2024-09-16T03:40:28.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.590918+0000", 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9299999999999992e-06 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.590971+0000", 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3015e-05 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.590978+0000", 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5420000000000002e-06 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.592194+0000", 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001216165 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.592227+0000", 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3101e-05 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.9s4 42/31 ECSubWrite(tid=32, reqid=client.4615.0:261, at_version=42'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.592543+0000", 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.9708254830000005, 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084283700000000003, 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.643 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.644 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.592543+0000", 2024-09-16T03:40:28.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.592543+0000", 2024-09-16T03:40:28.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.592543+0000", 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8999999999999996e-07 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.592695+0000", 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015141500000000001 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.592704+0000", 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7250000000000003e-06 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.592709+0000", 2024-09-16T03:40:28.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1220000000000003e-06 2024-09-16T03:40:28.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.592747+0000", 2024-09-16T03:40:28.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8513999999999998e-05 2024-09-16T03:40:28.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.592753+0000", 2024-09-16T03:40:28.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T03:40:28.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.593359+0000", 2024-09-16T03:40:28.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00060635799999999998 2024-09-16T03:40:28.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.593386+0000", 2024-09-16T03:40:28.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6183999999999999e-05 2024-09-16T03:40:28.649 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.649 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.649 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.650 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 42/31 ECSubWrite(tid=22, reqid=client.4615.0:266, at_version=42'13, trim_to=0'0, roll_forward_to=34'11))", 2024-09-16T03:40:28.650 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.624591+0000", 2024-09-16T03:40:28.650 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.9387768510000001, 2024-09-16T03:40:28.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012810390000000001, 2024-09-16T03:40:28.650 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.650 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.650 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.650 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.624591+0000", 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.624591+0000", 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.624592+0000", 2024-09-16T03:40:28.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5200000000000001e-07 2024-09-16T03:40:28.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.624916+0000", 2024-09-16T03:40:28.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00032438299999999997 2024-09-16T03:40:28.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.624927+0000", 2024-09-16T03:40:28.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1516e-05 2024-09-16T03:40:28.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.624936+0000", 2024-09-16T03:40:28.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7229999999999993e-06 2024-09-16T03:40:28.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.624999+0000", 2024-09-16T03:40:28.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2687000000000005e-05 2024-09-16T03:40:28.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.625007+0000", 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2160000000000006e-06 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.625841+0000", 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00083363899999999995 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.625872+0000", 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1723000000000001e-05 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.655 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 42/31 ECSubWrite(tid=23, reqid=client.4615.0:267, at_version=42'14, trim_to=0'0, roll_forward_to=34'11))", 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.627696+0000", 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.9356721569999999, 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001729756, 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.627696+0000", 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.627696+0000", 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.627697+0000", 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:40:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.627956+0000", 2024-09-16T03:40:28.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025896799999999999 2024-09-16T03:40:28.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.627967+0000", 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1694e-05 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.627977+0000", 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9159999999999996e-06 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.628023+0000", 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5243e-05 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.628028+0000", 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9229999999999999e-06 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629402+0000", 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001373174 2024-09-16T03:40:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629426+0000", 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4275999999999999e-05 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 42/31 ECSubWrite(tid=24, reqid=client.4615.0:268, at_version=42'15, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.629048+0000", 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.9343202100000001, 2024-09-16T03:40:28.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00069481899999999995, 2024-09-16T03:40:28.662 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.662 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.662 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.662 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629048+0000", 2024-09-16T03:40:28.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629048+0000", 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629049+0000", 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8599999999999998e-07 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629105+0000", 2024-09-16T03:40:28.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6243999999999999e-05 2024-09-16T03:40:28.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629116+0000", 2024-09-16T03:40:28.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1246999999999999e-05 2024-09-16T03:40:28.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629121+0000", 2024-09-16T03:40:28.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1959999999999999e-06 2024-09-16T03:40:28.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629162+0000", 2024-09-16T03:40:28.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0638e-05 2024-09-16T03:40:28.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629167+0000", 2024-09-16T03:40:28.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4099999999999999e-06 2024-09-16T03:40:28.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629724+0000", 2024-09-16T03:40:28.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00055667300000000004 2024-09-16T03:40:28.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.629743+0000", 2024-09-16T03:40:28.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8924999999999999e-05 2024-09-16T03:40:28.667 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.667 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.668 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.668 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 43/31 ECSubWrite(tid=28, reqid=client.4615.0:280, at_version=43'18, trim_to=0'0, roll_forward_to=42'16))", 2024-09-16T03:40:28.668 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:22.575479+0000", 2024-09-16T03:40:28.668 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.9878893919999996, 2024-09-16T03:40:28.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010616689999999999, 2024-09-16T03:40:28.669 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.669 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.669 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.669 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.575479+0000", 2024-09-16T03:40:28.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.575479+0000", 2024-09-16T03:40:28.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.575479+0000", 2024-09-16T03:40:28.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1699999999999999e-07 2024-09-16T03:40:28.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.575602+0000", 2024-09-16T03:40:28.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000123282 2024-09-16T03:40:28.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.575617+0000", 2024-09-16T03:40:28.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4047e-05 2024-09-16T03:40:28.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.575625+0000", 2024-09-16T03:40:28.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7129999999999994e-06 2024-09-16T03:40:28.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.575672+0000", 2024-09-16T03:40:28.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7074999999999999e-05 2024-09-16T03:40:28.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.575678+0000", 2024-09-16T03:40:28.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8919999999999997e-06 2024-09-16T03:40:28.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.576504+0000", 2024-09-16T03:40:28.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00082587299999999997 2024-09-16T03:40:28.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.576541+0000", 2024-09-16T03:40:28.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6470000000000001e-05 2024-09-16T03:40:28.675 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.675 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.675 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.676 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 43/31 ECSubWrite(tid=29, reqid=client.4615.0:280, at_version=0'0, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:28.676 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:22.577599+0000", 2024-09-16T03:40:28.676 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.9857691500000003, 2024-09-16T03:40:28.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00061223099999999997, 2024-09-16T03:40:28.676 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:28.676 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:28.677 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:28.677 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:28.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:28.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.577599+0000", 2024-09-16T03:40:28.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:28.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.577599+0000", 2024-09-16T03:40:28.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:28.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:28.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.577600+0000", 2024-09-16T03:40:28.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9599999999999998e-07 2024-09-16T03:40:28.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:28.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.577613+0000", 2024-09-16T03:40:28.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2952999999999999e-05 2024-09-16T03:40:28.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.577627+0000", 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4022999999999999e-05 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.577637+0000", 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9420000000000007e-06 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.577679+0000", 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2150999999999997e-05 2024-09-16T03:40:28.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.577685+0000", 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2689999999999996e-06 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.578190+0000", 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00050463799999999996 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:28.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:22.578211+0000", 2024-09-16T03:40:28.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1458999999999999e-05 2024-09-16T03:40:28.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.682 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:28.682 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:28.682 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:28.683 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:28.682+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:28.684 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:40:28.685 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:28.685 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:28.685 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:28.685 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:28.686 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:28.686 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:28.686 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:28.687 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:28.687 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:28.687 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:28.687 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:28.688 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:28.692 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:28.693 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:28.709 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:28.709 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:28.726 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:28.726 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:28.742 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:28.743 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:28.760 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:28.760 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:28.777 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:28.777 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:28.784 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:28.782+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:28.794 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:28.794 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:28.794 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:28.795 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:28.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:28.799+0000 7f2925be1640 0 --2- 172.21.15.76:0/1302519138 >> v2:172.21.15.76:3300/0 conn(0x7f2920152240 0x7f29200c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T03:40:28.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:40:28.815 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:28.816 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:28.832 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:28.832 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:28.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:28.831+0000 7f0566d9d640 0 -- 172.21.15.76:0/767291382 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f0568223120 msgr2=0x7f0568225510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T03:40:28.885 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:28.884+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:28.920 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:28.920 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:28.929 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:40:28.986 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:28.983+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:29.050 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:29.051 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:29.074 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:29.075 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:29.084 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:40:29.086 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:29.084+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:29.187 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:29.186+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:29.230 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:29.230 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:29.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:40:29.288 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:29.286+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:29.383 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:29.383 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:29.389 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:29.387+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:29.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:40:29.479 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T03:40:29.483 INFO:tasks.rados.rados.0.smithi121.stdout:124: done (4 left) 2024-09-16T03:40:29.483 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 11 current snap is 11 2024-09-16T03:40:29.483 INFO:tasks.rados.rados.0.smithi121.stdout:118: finishing write tid 2 to smithi12138290-30 2024-09-16T03:40:29.483 INFO:tasks.rados.rados.0.smithi121.stdout:125: seq_num 77 ranges {688128=524288,1212416=524288,1736704=245760} 2024-09-16T03:40:29.489 INFO:tasks.rados.rados.0.smithi121.stdout:125: writing smithi12138290-11 from 688128 to 1212416 tid 1 2024-09-16T03:40:29.489 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:29.487+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:29.492 INFO:tasks.rados.rados.0.smithi121.stdout:125: writing smithi12138290-11 from 1212416 to 1736704 tid 2 2024-09-16T03:40:29.494 INFO:tasks.rados.rados.0.smithi121.stdout:125: writing smithi12138290-11 from 1736704 to 1982464 tid 3 2024-09-16T03:40:29.494 INFO:tasks.rados.rados.0.smithi121.stdout:118: finishing write tid 3 to smithi12138290-30 2024-09-16T03:40:29.494 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 34 v 14 (ObjNum 66 snap 3 seq_num 66) dirty exists 2024-09-16T03:40:29.494 INFO:tasks.rados.rados.0.smithi121.stdout:122: finishing rollback tid 0 to smithi12138290-3 2024-09-16T03:40:29.494 INFO:tasks.rados.rados.0.smithi121.stdout:122: finishing rollback tid 1 to smithi12138290-3 2024-09-16T03:40:29.494 INFO:tasks.rados.rados.0.smithi121.stdout:121: finishing copy_from racing read to smithi12138290-1 2024-09-16T03:40:29.494 INFO:tasks.rados.rados.0.smithi121.stdout:122: finishing rollback tid 2 to smithi12138290-3 2024-09-16T03:40:29.494 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 3 v 18 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:121: finishing copy_from to smithi12138290-1 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 1 v 4 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:118: finishing write tid 4 to smithi12138290-30 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:118: oid 30 updating version 0 to 23 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:118: oid 30 version 23 is already newer than 20 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:118: oid 30 updating version 23 to 24 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 30 v 24 (ObjNum 75 snap 11 seq_num 75) dirty exists 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:118: left oid 30 (ObjNum 75 snap 11 seq_num 75) 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:118: done (4 left) 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:120: done (3 left) 2024-09-16T03:40:29.495 INFO:tasks.rados.rados.0.smithi121.stdout:121: done (2 left) 2024-09-16T03:40:29.496 INFO:tasks.rados.rados.0.smithi121.stdout:122: done (1 left) 2024-09-16T03:40:29.496 INFO:tasks.rados.rados.0.smithi121.stdout:126: delete oid 26 current snap is 11 2024-09-16T03:40:29.500 INFO:tasks.rados.rados.0.smithi121.stdout:126: done (1 left) 2024-09-16T03:40:29.500 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 4 current snap is 11 2024-09-16T03:40:29.500 INFO:tasks.rados.rados.0.smithi121.stdout:127: seq_num 79 ranges {0=573440,573440=425984,999424=344064} 2024-09-16T03:40:29.504 INFO:tasks.rados.rados.0.smithi121.stdout:127: writing smithi12138290-4 from 0 to 573440 tid 1 2024-09-16T03:40:29.507 INFO:tasks.rados.rados.0.smithi121.stdout:127: writing smithi12138290-4 from 573440 to 999424 tid 2 2024-09-16T03:40:29.509 INFO:tasks.rados.rados.0.smithi121.stdout:127: writing smithi12138290-4 from 999424 to 1343488 tid 3 2024-09-16T03:40:29.509 INFO:tasks.rados.rados.0.smithi121.stdout:128: snap_remove snap 10 2024-09-16T03:40:29.510 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:29.510 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:29.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:40:29.590 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:29.588+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:29.631 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:29.632 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:29.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:40:29.691 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:29.689+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:29.743 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:29.743 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:29.752 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:40:29.792 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:29.790+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:29.855 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:29.855 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:29.864 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:40:29.892 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:29.890+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:29.967 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:29.967 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:29.976 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:40:30.019 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:30.018+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:30.079 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:30.079 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:30.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:40:30.094 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:30.092+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:30.194 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:30.193+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:30.232 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:30.232 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:30.241 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:40:30.295 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:30.294+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:30.344 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:30.345 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:30.353 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:40:30.396 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:30.395+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:30.455 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:30.455 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:30.464 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:40:30.486 INFO:tasks.rados.rados.0.smithi121.stdout:128: done (2 left) 2024-09-16T03:40:30.486 INFO:tasks.rados.rados.0.smithi121.stdout:129: read oid 49 snap -1 2024-09-16T03:40:30.486 INFO:tasks.rados.rados.0.smithi121.stdout:129: expect deleted 2024-09-16T03:40:30.486 INFO:tasks.rados.rados.0.smithi121.stdout:130: snap_create 2024-09-16T03:40:30.486 INFO:tasks.rados.rados.0.smithi121.stdout:125: finishing write tid 1 to smithi12138290-11 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:125: finishing write tid 2 to smithi12138290-11 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:125: finishing write tid 3 to smithi12138290-11 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:125: finishing write tid 4 to smithi12138290-11 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:125: finishing write tid 5 to smithi12138290-11 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:125: oid 11 updating version 0 to 29 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:125: oid 11 updating version 29 to 31 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:125: oid 11 version 31 is already newer than 26 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:125: oid 11 version 31 is already newer than 30 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 11 v 31 (ObjNum 77 snap 11 seq_num 77) dirty exists 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:125: left oid 11 (ObjNum 77 snap 11 seq_num 77) 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:127: finishing write tid 1 to smithi12138290-4 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:127: finishing write tid 2 to smithi12138290-4 2024-09-16T03:40:30.487 INFO:tasks.rados.rados.0.smithi121.stdout:127: finishing write tid 3 to smithi12138290-4 2024-09-16T03:40:30.488 INFO:tasks.rados.rados.0.smithi121.stdout:127: finishing write tid 4 to smithi12138290-4 2024-09-16T03:40:30.488 INFO:tasks.rados.rados.0.smithi121.stdout:127: finishing write tid 5 to smithi12138290-4 2024-09-16T03:40:30.488 INFO:tasks.rados.rados.0.smithi121.stdout:127: oid 4 updating version 0 to 25 2024-09-16T03:40:30.488 INFO:tasks.rados.rados.0.smithi121.stdout:127: oid 4 updating version 25 to 27 2024-09-16T03:40:30.488 INFO:tasks.rados.rados.0.smithi121.stdout:127: oid 4 updating version 27 to 28 2024-09-16T03:40:30.488 INFO:tasks.rados.rados.0.smithi121.stdout:127: oid 4 version 28 is already newer than 26 2024-09-16T03:40:30.488 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 4 v 28 (ObjNum 79 snap 11 seq_num 79) dirty exists 2024-09-16T03:40:30.488 INFO:tasks.rados.rados.0.smithi121.stdout:127: left oid 4 (ObjNum 79 snap 11 seq_num 79) 2024-09-16T03:40:30.497 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:30.496+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:30.571 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:30.571 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:30.579 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:40:30.598 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:30.596+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:30.683 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:30.683 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:30.692 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:40:30.699 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:30.697+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:30.796 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:30.796 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:30.799 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:30.798+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:30.804 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:40:30.900 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:30.899+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:30.908 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:30.908 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:30.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:40:31.001 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:30.998+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:31.032 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:31.032 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:31.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:40:31.062 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:31.101 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:31.100+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:31.144 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:31.144 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:31.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:40:31.194 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:31.202 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:31.201+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:31.211 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:31.211 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:31.228 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:31.228 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:31.245 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:31.245 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:31.258 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 37/31 ECSubWrite(tid=60, reqid=client.4615.0:209, at_version=37'22, trim_to=0'0, roll_forward_to=34'20))", 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.526020+0000", 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 14.729825849999999, 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001541396, 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526020+0000", 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526020+0000", 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526020+0000", 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7000000000000001e-07 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526048+0000", 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8382999999999999e-05 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526061+0000", 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3047000000000001e-05 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526073+0000", 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1742e-05 2024-09-16T03:40:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526150+0000", 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7155000000000002e-05 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526164+0000", 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3661e-05 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527519+0000", 2024-09-16T03:40:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001355476 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527561+0000", 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1662000000000002e-05 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=25, reqid=client.4615.0:217, at_version=37'7, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.555333+0000", 2024-09-16T03:40:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 14.700512043, 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010782770000000001, 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555333+0000", 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555333+0000", 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555334+0000", 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4299999999999999e-07 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555358+0000", 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4924999999999999e-05 2024-09-16T03:40:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555367+0000", 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4619999999999996e-06 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555374+0000", 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4390000000000003e-06 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555443+0000", 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8359000000000001e-05 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555452+0000", 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1989999999999995e-06 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556371+0000", 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000918642 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556412+0000", 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1007999999999998e-05 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=27, reqid=client.4615.0:219, at_version=37'9, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.557379+0000", 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 14.698466230999999, 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002658929, 2024-09-16T03:40:31.268 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557379+0000", 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557379+0000", 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557380+0000", 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6100000000000001e-07 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557395+0000", 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5763000000000001e-05 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557408+0000", 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2183e-05 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557415+0000", 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.622e-06 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557481+0000", 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5617999999999994e-05 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557490+0000", 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0280000000000006e-06 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.560006+0000", 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025159779999999999 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.560038+0000", 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2175999999999998e-05 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 37/31 ECSubWrite(tid=14, reqid=client.4615.0:220, at_version=37'10, trim_to=0'0, roll_forward_to=34'8))", 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.571470+0000", 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 14.684375035, 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00087942399999999996, 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571470+0000", 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571470+0000", 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571470+0000", 2024-09-16T03:40:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.08e-07 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571485+0000", 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4426999999999999e-05 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571495+0000", 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571501+0000", 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7610000000000004e-06 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571554+0000", 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2268999999999997e-05 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.571559+0000", 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.699e-06 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.572323+0000", 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076385799999999996 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.572350+0000", 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6684000000000001e-05 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 37/31 ECSubWrite(tid=15, reqid=client.4615.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.573070+0000", 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 14.682775606, 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020277009999999998, 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573070+0000", 2024-09-16T03:40:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573070+0000", 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573070+0000", 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573078+0000", 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1919999999999999e-06 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573105+0000", 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7830999999999998e-05 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573112+0000", 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6499999999999999e-06 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573189+0000", 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7373999999999995e-05 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.573196+0000", 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.575065+0000", 2024-09-16T03:40:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001869419 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.575097+0000", 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2014999999999998e-05 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 38/31 ECSubWrite(tid=62, reqid=client.4615.0:225, at_version=38'9, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.542570+0000", 2024-09-16T03:40:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.713275727999999, 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018344209999999999, 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.542570+0000", 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.542570+0000", 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.542570+0000", 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4000000000000001e-07 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.543309+0000", 2024-09-16T03:40:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073942400000000003 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.543321+0000", 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1773999999999999e-05 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.543330+0000", 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.543392+0000", 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2198000000000003e-05 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.543399+0000", 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4490000000000002e-06 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544371+0000", 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00097108700000000004 2024-09-16T03:40:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544404+0000", 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3552e-05 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 38/31 ECSubWrite(tid=63, reqid=client.4615.0:226, at_version=38'10, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.544027+0000", 2024-09-16T03:40:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.711818579999999, 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001061338, 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544027+0000", 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544027+0000", 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544028+0000", 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1000000000000001e-06 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544393+0000", 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00036505000000000001 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544401+0000", 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3489999999999992e-06 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544409+0000", 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7749999999999993e-06 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544447+0000", 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8164000000000003e-05 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.544453+0000", 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4009999999999996e-06 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.545066+0000", 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00061321499999999996 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.545088+0000", 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2283999999999999e-05 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 38/31 ECSubWrite(tid=20, reqid=client.4615.0:229, at_version=38'11, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:17.553469+0000", 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.702376646999999, 2024-09-16T03:40:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023386560000000002, 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553469+0000", 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553469+0000", 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553469+0000", 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.01e-07 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.553994+0000", 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00052489500000000005 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554009+0000", 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4907e-05 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554020+0000", 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0691e-05 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554091+0000", 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1778000000000003e-05 2024-09-16T03:40:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.554098+0000", 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.331e-06 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555769+0000", 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001671723 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:17.555807+0000", 2024-09-16T03:40:31.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7929999999999998e-05 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 42/31 ECSubWrite(tid=22, reqid=osd.8.0:22, at_version=42'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.552469+0000", 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.7033764799999993, 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00163339, 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.552469+0000", 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.552469+0000", 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.552469+0000", 2024-09-16T03:40:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.91e-07 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.552495+0000", 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5316999999999998e-05 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.552508+0000", 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3108000000000001e-05 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.552518+0000", 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0142e-05 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.552605+0000", 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7348999999999993e-05 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.552618+0000", 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3217000000000001e-05 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.554057+0000", 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001438677 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.554102+0000", 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5289e-05 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 42/31 ECSubWrite(tid=23, reqid=osd.8.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.555043+0000", 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.7008025900000003, 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010750009999999999, 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555043+0000", 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555043+0000", 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555044+0000", 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.06e-06 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555060+0000", 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6433e-05 2024-09-16T03:40:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555195+0000", 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000135086 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555206+0000", 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0478e-05 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555264+0000", 2024-09-16T03:40:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7788000000000003e-05 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555274+0000", 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9720000000000004e-06 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.556077+0000", 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080311100000000002 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.556118+0000", 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1072999999999998e-05 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 42/31 ECSubWrite(tid=73, reqid=osd.0.0:73, at_version=42'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.555861+0000", 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.6999847389999996, 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002093999, 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555861+0000", 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555861+0000", 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555861+0000", 2024-09-16T03:40:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5900000000000003e-07 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555897+0000", 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6442000000000001e-05 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555909+0000", 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1389000000000001e-05 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555918+0000", 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8769999999999999e-06 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.555993+0000", 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5291999999999997e-05 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.556008+0000", 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4776999999999999e-05 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.557913+0000", 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001904736 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.557955+0000", 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2126999999999998e-05 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 42/31 ECSubWrite(tid=75, reqid=client.4615.0:256, at_version=42'25, trim_to=0'0, roll_forward_to=41'23))", 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.579904+0000", 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.675941516, 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001393041, 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.579904+0000", 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.579904+0000", 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.579906+0000", 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8470000000000001e-06 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.579975+0000", 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9278999999999999e-05 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.580003+0000", 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8415000000000002e-05 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.580018+0000", 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4324e-05 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.580219+0000", 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000201461 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.580247+0000", 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7860000000000001e-05 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.581257+0000", 2024-09-16T03:40:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010099589999999999 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.581297+0000", 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9895999999999999e-05 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=77, reqid=client.4615.0:275, at_version=43'15, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.568085+0000", 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 8.6877608619999993, 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020462520000000001, 2024-09-16T03:40:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.568085+0000", 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.568085+0000", 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.568085+0000", 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.36e-07 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.568414+0000", 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00032937800000000002 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.568428+0000", 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4134000000000001e-05 2024-09-16T03:40:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.568437+0000", 2024-09-16T03:40:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8759999999999994e-06 2024-09-16T03:40:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.568532+0000", 2024-09-16T03:40:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4891999999999999e-05 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.568546+0000", 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4249e-05 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.570085+0000", 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015384979999999999 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.570131+0000", 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6088999999999999e-05 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=78, reqid=client.4615.0:276, at_version=43'16, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.570496+0000", 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 8.6853490799999999, 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0016787849999999999, 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.570496+0000", 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.570496+0000", 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.570497+0000", 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8999999999999995e-07 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571102+0000", 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00060431899999999997 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571115+0000", 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3216e-05 2024-09-16T03:40:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571122+0000", 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3549999999999999e-06 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571180+0000", 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8195000000000001e-05 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571187+0000", 2024-09-16T03:40:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1470000000000004e-06 2024-09-16T03:40:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.572144+0000", 2024-09-16T03:40:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00095611700000000003 2024-09-16T03:40:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.572175+0000", 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1545999999999999e-05 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=79, reqid=client.4615.0:277, at_version=43'17, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.571700+0000", 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 8.6841455889999999, 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00099067800000000004, 2024-09-16T03:40:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571700+0000", 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571700+0000", 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571701+0000", 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.09e-07 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571725+0000", 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3845999999999998e-05 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571737+0000", 2024-09-16T03:40:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2274e-05 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571745+0000", 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7009999999999996e-06 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571793+0000", 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8723000000000001e-05 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571800+0000", 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0480000000000001e-06 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.572663+0000", 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00086240899999999996 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.572690+0000", 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7767999999999999e-05 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 49/31 ECSubWrite(tid=83, reqid=osd.0.0:83, at_version=49'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491302+0000", 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.7645432369999998, 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001679133, 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491302+0000", 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491302+0000", 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491302+0000", 2024-09-16T03:40:31.332 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8099999999999999e-07 2024-09-16T03:40:31.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491342+0000", 2024-09-16T03:40:31.332 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0077999999999998e-05 2024-09-16T03:40:31.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491357+0000", 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4303e-05 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491369+0000", 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2207e-05 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491477+0000", 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010841 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491489+0000", 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1243999999999999e-05 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492934+0000", 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001445207 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492981+0000", 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7503000000000002e-05 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=30, reqid=client.4615.0:314, at_version=49'20, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.76227467, 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0049994970000000003, 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6100000000000002e-07 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493601+0000", 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9561999999999999e-05 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493614+0000", 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3421000000000001e-05 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493623+0000", 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4050000000000006e-06 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493705+0000", 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1500000000000002e-05 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493719+0000", 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3866000000000001e-05 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.498525+0000", 2024-09-16T03:40:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048063230000000004 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.498570+0000", 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5058999999999997e-05 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=31, reqid=osd.8.0:31, at_version=49'22, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.7603878100000001, 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059536579999999997, 2024-09-16T03:40:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495459+0000", 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3799999999999996e-07 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495484+0000", 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5601000000000001e-05 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495505+0000", 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0460999999999999e-05 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495514+0000", 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1390000000000001e-06 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495592+0000", 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7877e-05 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495607+0000", 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5576000000000001e-05 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501362+0000", 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057546280000000003 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501411+0000", 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9438000000000003e-05 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 49/31 ECSubWrite(tid=84, reqid=osd.0.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=49'20))", 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.497586+0000", 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.7582597, 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041019860000000002, 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497586+0000", 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497586+0000", 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497587+0000", 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0100000000000001e-06 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497598+0000", 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1365999999999999e-05 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497613+0000", 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4752e-05 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497619+0000", 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5980000000000002e-06 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497688+0000", 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8973999999999994e-05 2024-09-16T03:40:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497698+0000", 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5550000000000008e-06 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501634+0000", 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039355780000000003 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501688+0000", 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4153000000000001e-05 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=32, reqid=osd.8.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=49'22))", 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.502382+0000", 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.7534635810000001, 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010002590000000001, 2024-09-16T03:40:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502382+0000", 2024-09-16T03:40:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502382+0000", 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502383+0000", 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5099999999999998e-07 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502395+0000", 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1929999999999999e-05 2024-09-16T03:40:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502428+0000", 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3657e-05 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502439+0000", 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0463e-05 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502529+0000", 2024-09-16T03:40:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9912000000000002e-05 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502538+0000", 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6299999999999993e-06 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503355+0000", 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00081717200000000002 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503382+0000", 2024-09-16T03:40:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6644000000000001e-05 2024-09-16T03:40:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.356 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:31.357 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:31.358 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:31.358 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:31.359 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:31.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:40:31.359 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:31.360 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:31.360 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:31.360 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:31.361 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:31.361 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:31.361 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:31.362 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:31.362 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:31.356+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:31.362 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:31.362 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:31.375 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:31.375 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:31.390 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:31.391 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:31.407 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:31.407 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:31.423 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:31.423 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:31.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:31.422+0000 7fe510802640 0 -- 172.21.15.76:0/4059778493 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7fe50c2209b0 msgr2=0x7fe50c222da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:31.458 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:31.456+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:31.472 INFO:tasks.rados.rados.0.smithi121.stdout:125: done (3 left) 2024-09-16T03:40:31.472 INFO:tasks.rados.rados.0.smithi121.stdout:127: done (2 left) 2024-09-16T03:40:31.472 INFO:tasks.rados.rados.0.smithi121.stdout:129: done (1 left) 2024-09-16T03:40:31.472 INFO:tasks.rados.rados.0.smithi121.stdout:130: done (0 left) 2024-09-16T03:40:31.472 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 39 current snap is 12 2024-09-16T03:40:31.473 INFO:tasks.rados.rados.0.smithi121.stdout:131: seq_num 80 ranges {0=589824,589824=573440,1163264=196608} 2024-09-16T03:40:31.475 INFO:tasks.rados.rados.0.smithi121.stdout:131: writing smithi12138290-39 from 0 to 589824 tid 1 2024-09-16T03:40:31.478 INFO:tasks.rados.rados.0.smithi121.stdout:131: writing smithi12138290-39 from 589824 to 1163264 tid 2 2024-09-16T03:40:31.480 INFO:tasks.rados.rados.0.smithi121.stdout:131: writing smithi12138290-39 from 1163264 to 1359872 tid 3 2024-09-16T03:40:31.480 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 30 current snap is 12 2024-09-16T03:40:31.480 INFO:tasks.rados.rados.0.smithi121.stdout:132: seq_num 81 ranges {1753088=606208,2359296=720896,3080192=49152} 2024-09-16T03:40:31.493 INFO:tasks.rados.rados.0.smithi121.stdout:132: writing smithi12138290-30 from 1753088 to 2359296 tid 1 2024-09-16T03:40:31.497 INFO:tasks.rados.rados.0.smithi121.stdout:132: writing smithi12138290-30 from 2359296 to 3080192 tid 2 2024-09-16T03:40:31.498 INFO:tasks.rados.rados.0.smithi121.stdout:132: writing smithi12138290-30 from 3080192 to 3129344 tid 3 2024-09-16T03:40:31.498 INFO:tasks.rados.rados.0.smithi121.stdout:133: rollback oid 12 current snap is 12 2024-09-16T03:40:31.498 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 12 to 1 2024-09-16T03:40:31.498 INFO:tasks.rados.rados.0.smithi121.stdout:131: finishing write tid 1 to smithi12138290-39 2024-09-16T03:40:31.498 INFO:tasks.rados.rados.0.smithi121.stdout:131: finishing write tid 2 to smithi12138290-39 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:131: finishing write tid 3 to smithi12138290-39 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:131: finishing write tid 4 to smithi12138290-39 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:131: finishing write tid 5 to smithi12138290-39 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:131: oid 39 updating version 0 to 15 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:131: oid 39 version 15 is already newer than 14 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:131: oid 39 version 15 is already newer than 13 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:131: oid 39 version 15 is already newer than 12 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 39 v 15 (ObjNum 80 snap 12 seq_num 80) dirty exists 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:131: left oid 39 (ObjNum 80 snap 12 seq_num 80) 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:131: done (2 left) 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:134: copy_from oid 24 from oid 8 current snap is 12 2024-09-16T03:40:31.499 INFO:tasks.rados.rados.0.smithi121.stdout:135: snap_remove snap 8 2024-09-16T03:40:31.500 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:40:31.511 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:31.511 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:31.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.511 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 52/31 ECSubWrite(tid=39, reqid=client.4615.0:344, at_version=0'0, trim_to=0'0, roll_forward_to=52'18))", 2024-09-16T03:40:31.511 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.507734+0000", 2024-09-16T03:40:31.511 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0019765669999999998, 2024-09-16T03:40:31.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001990852, 2024-09-16T03:40:31.511 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.511 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.507734+0000", 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.507734+0000", 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.507734+0000", 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5499999999999997e-07 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.507744+0000", 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0599999999999997e-06 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.507753+0000", 2024-09-16T03:40:31.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3209999999999994e-06 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.507758+0000", 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1510000000000003e-06 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.507800+0000", 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1727000000000002e-05 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.507810+0000", 2024-09-16T03:40:31.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9750000000000002e-06 2024-09-16T03:40:31.515 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.515 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.515 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.515 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.515 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-16T03:40:31.515 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 1 2024-09-16T03:40:31.515 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:31.521 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:40:31.566 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:31.565+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:31.624 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:31.624 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:31.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:40:31.641 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:31.641 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:31.660 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:31.659+0000 7fc5879d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33192) UID: 0 2024-09-16T03:40:31.738 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:31.738 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:31.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:31.738 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:31.738 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.738 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:268 3.1s0 3:82a78c8d:::smithi12138290-1:head [append 0~196608] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:31.738 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.627532+0000", 2024-09-16T03:40:31.738 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 10.107670281000001, 2024-09-16T03:40:31.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075158530000000003, 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 268 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627532+0000", 2024-09-16T03:40:31.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627532+0000", 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627533+0000", 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6599999999999995e-07 2024-09-16T03:40:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627834+0000", 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00030093699999999998 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627844+0000", 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0478e-05 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627850+0000", 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7910000000000001e-06 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.628455+0000", 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00060544400000000001 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.628522+0000", 2024-09-16T03:40:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6670999999999999e-05 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.629026+0000", 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050412799999999995 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630252+0000", 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001225711 2024-09-16T03:40:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.634935+0000", 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046835130000000003 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635048+0000", 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000112414 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:269 3.1s0 3:82a78c8d:::smithi12138290-1:head [setxattr _header (55) in=7b] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.627861+0000", 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 10.107341579, 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073128849999999999, 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 269 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627861+0000", 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627861+0000", 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627861+0000", 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8100000000000003e-07 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627864+0000", 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8490000000000002e-06 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627870+0000", 2024-09-16T03:40:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0299999999999999e-06 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627873+0000", 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2440000000000002e-06 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.629522+0000", 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001648634 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.629585+0000", 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2908000000000001e-05 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.629951+0000", 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00036612199999999998 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630681+0000", 2024-09-16T03:40:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073026799999999995 2024-09-16T03:40:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635120+0000", 2024-09-16T03:40:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044385550000000003 2024-09-16T03:40:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635174+0000", 2024-09-16T03:40:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3693999999999998e-05 2024-09-16T03:40:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:270 3.1s0 3:82a78c8d:::smithi12138290-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 10.107324493, 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086784840000000002, 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 270 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8299999999999998e-07 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627880+0000", 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.716e-06 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627895+0000", 2024-09-16T03:40:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4904e-05 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627900+0000", 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4990000000000002e-06 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630550+0000", 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0026501960000000001 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630645+0000", 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4790999999999994e-05 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635427+0000", 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047821740000000001 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635478+0000", 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0603000000000003e-05 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.636556+0000", 2024-09-16T03:40:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001078318 2024-09-16T03:40:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:280 3.1s0 3:8df00818:::smithi12138290-14:head [copy-from ver 2] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.569179+0000", 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.1660233820000006, 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080900399999999997, 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 280 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569179+0000", 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569179+0000", 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569181+0000", 2024-09-16T03:40:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3970000000000001e-06 2024-09-16T03:40:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569193+0000", 2024-09-16T03:40:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1744e-05 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569210+0000", 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7286e-05 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569223+0000", 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2852e-05 2024-09-16T03:40:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569304+0000", 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0815000000000005e-05 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569310+0000", 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9800000000000003e-06 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571309+0000", 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001999158 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571423+0000", 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011391399999999999 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575663+0000", 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042401619999999996 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.576691+0000", 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010281489999999999 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577144+0000", 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00045261500000000002 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577269+0000", 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012496799999999999 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:281 3.1s0 3:8df00818:::smithi12138290-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.569246+0000", 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.1659560950000003, 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073290630000000002, 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 281 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569246+0000", 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569246+0000", 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569247+0000", 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2600000000000002e-07 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569251+0000", 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3320000000000002e-06 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569264+0000", 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2839e-05 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569270+0000", 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8039999999999998e-06 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569378+0000", 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010869700000000001 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569385+0000", 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8569999999999997e-06 2024-09-16T03:40:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571608+0000", 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022225389999999999 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571658+0000", 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9840999999999999e-05 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.576351+0000", 2024-09-16T03:40:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046931289999999999 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.576453+0000", 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000101808 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.576575+0000", 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012259099999999999 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:301 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~491520] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.2523441480000002, 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013883662999999999, 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 301 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482860+0000", 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.911e-06 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483803+0000", 2024-09-16T03:40:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00094328700000000001 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483820+0000", 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6249e-05 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483828+0000", 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.704e-06 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483895+0000", 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6839000000000002e-05 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483902+0000", 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.485797+0000", 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018946049999999999 2024-09-16T03:40:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.485916+0000", 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011867 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487208+0000", 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012921110000000001 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493186+0000", 2024-09-16T03:40:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059786010000000001 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496639+0000", 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034530749999999999 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496742+0000", 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102394 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:302 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~770048] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.2480648749999999, 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014729638999999999, 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 302 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:40:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487139+0000", 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.378e-06 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488116+0000", 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00097674700000000003 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488127+0000", 2024-09-16T03:40:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1817999999999999e-05 2024-09-16T03:40:31.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488135+0000", 2024-09-16T03:40:31.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6620000000000005e-06 2024-09-16T03:40:31.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489443+0000", 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001308303 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489564+0000", 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000120664 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490525+0000", 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00096115100000000004 2024-09-16T03:40:31.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493256+0000", 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027313429999999998 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501784+0000", 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0085277840000000001 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501867+0000", 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2788999999999996e-05 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:303 3.1s0 3:8a735ecd:::smithi12138290-30:head [setxattr _header (57) in=7b] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.2470533389999998, 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013931318, 2024-09-16T03:40:31.787 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 303 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:40:31.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488150+0000", 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000001e-07 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488152+0000", 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.509e-06 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488157+0000", 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5739999999999997e-06 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488160+0000", 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.349e-06 2024-09-16T03:40:31.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491536+0000", 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033756799999999998 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491638+0000", 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010186799999999999 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492139+0000", 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050129999999999999 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493257+0000", 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001118551 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501989+0000", 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087319819999999992 2024-09-16T03:40:31.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502080+0000", 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0827000000000003e-05 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:304 3.1s0 3:8a735ecd:::smithi12138290-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.2470363610000001, 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.024338029000000001, 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.793 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 304 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.53e-07 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488169+0000", 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3309999999999998e-06 2024-09-16T03:40:31.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488173+0000", 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488176+0000", 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0539999999999999e-06 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492435+0000", 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042594399999999998 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492534+0000", 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8558000000000001e-05 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505269+0000", 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012735076 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505321+0000", 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1977000000000001e-05 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.512504+0000", 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0071833060000000004 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:309 3.1s0 3:82a78c8d:::smithi12138290-1:head [copy-from ver 4] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.244376908, 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020962743999999998, 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 309 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490827+0000", 2024-09-16T03:40:31.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490832+0000", 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4550000000000003e-06 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490845+0000", 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2951e-05 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490873+0000", 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7566999999999999e-05 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492571+0000", 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001698359 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492650+0000", 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8858999999999998e-05 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503971+0000", 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011320805 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511333+0000", 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073624160000000001 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511735+0000", 2024-09-16T03:40:31.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00040226999999999998 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511788+0000", 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2757999999999999e-05 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:310 3.1s0 3:82a78c8d:::smithi12138290-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:40:31.804 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.2443109290000001, 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014337858, 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 310 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490892+0000", 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.37e-07 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490896+0000", 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3710000000000002e-06 2024-09-16T03:40:31.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490932+0000", 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5796000000000001e-05 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490941+0000", 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8589999999999994e-06 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493109+0000", 2024-09-16T03:40:31.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021679310000000001 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493167+0000", 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7572999999999999e-05 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505077+0000", 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011909968999999999 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505164+0000", 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7409000000000003e-05 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505229+0000", 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5213000000000001e-05 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s1 49/31 ECSubWrite(tid=30, reqid=client.4615.0:314, at_version=49'20, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.493624+0000", 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.2415783930000002, 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0074256740000000002, 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493624+0000", 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493624+0000", 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493624+0000", 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3099999999999998e-07 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493650+0000", 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5216e-05 2024-09-16T03:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493660+0000", 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0088999999999999e-05 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493669+0000", 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1139999999999995e-06 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493736+0000", 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6883000000000003e-05 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493745+0000", 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5540000000000003e-06 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501011+0000", 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00726627 2024-09-16T03:40:31.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501050+0000", 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8117000000000001e-05 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=22, reqid=client.4615.0:308, at_version=49'14, trim_to=0'0, roll_forward_to=42'12))", 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.494598+0000", 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.240604179, 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084633480000000007, 2024-09-16T03:40:31.814 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494598+0000", 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494598+0000", 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494599+0000", 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6899999999999998e-07 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494838+0000", 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000239409 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494848+0000", 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8859999999999999e-06 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494855+0000", 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9739999999999996e-06 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494918+0000", 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3478000000000004e-05 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494925+0000", 2024-09-16T03:40:31.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1940000000000003e-06 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503029+0000", 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081040469999999996 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503061+0000", 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2891000000000001e-05 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.818 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 49/31 ECSubWriteReply(tid=34, last_complete=49'24, committed=1, applied=1))", 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.503107+0000", 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.2320955229999999, 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081770950000000005, 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503107+0000", 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503107+0000", 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503108+0000", 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.61e-07 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503112+0000", 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9239999999999998e-06 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503122+0000", 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4599999999999992e-06 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503129+0000", 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6540000000000002e-06 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511256+0000", 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081276979999999992 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511284+0000", 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7597999999999999e-05 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.822 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 49/31 ECSubWriteReply(tid=34, last_complete=49'24, committed=1, applied=1))", 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.503421+0000", 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.2317815219999999, 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078850580000000003, 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503421+0000", 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503421+0000", 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503422+0000", 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503426+0000", 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4689999999999999e-06 2024-09-16T03:40:31.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503439+0000", 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2802e-05 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503447+0000", 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0669999999999992e-06 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511292+0000", 2024-09-16T03:40:31.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078452020000000008 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511306+0000", 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3689000000000001e-05 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 49/31 ECSubWriteReply(tid=34, last_complete=49'24, committed=1, applied=1))", 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.231728537, 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00786215, 2024-09-16T03:40:31.826 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3099999999999998e-07 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503477+0000", 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1520000000000001e-06 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503488+0000", 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0387000000000001e-05 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503494+0000", 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9850000000000003e-06 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511323+0000", 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078294260000000004 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511336+0000", 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2669e-05 2024-09-16T03:40:31.829 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:337 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~606208] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.492469+0000", 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.24273331500000001, 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0083634099999999999, 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 337 2024-09-16T03:40:31.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.492469+0000", 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.492469+0000", 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.492470+0000", 2024-09-16T03:40:31.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.511e-06 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493117+0000", 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00064688400000000002 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493129+0000", 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.203e-05 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493135+0000", 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8139999999999997e-06 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493197+0000", 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2141e-05 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493305+0000", 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107426 2024-09-16T03:40:31.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.494433+0000", 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001128471 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496206+0000", 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001772608 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500752+0000", 2024-09-16T03:40:31.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045464759999999998 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500832+0000", 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0049000000000006e-05 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:338 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~720896] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.496989+0000", 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.238212798, 2024-09-16T03:40:31.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0077073139999999998, 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 338 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496989+0000", 2024-09-16T03:40:31.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496989+0000", 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496991+0000", 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0449999999999999e-06 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497710+0000", 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00071904 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497719+0000", 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3409999999999992e-06 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497723+0000", 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4610000000000001e-06 2024-09-16T03:40:31.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497761+0000", 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7469000000000003e-05 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497832+0000", 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1612999999999994e-05 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.498416+0000", 2024-09-16T03:40:31.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00058368299999999999 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.501013+0000", 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025971829999999999 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.504614+0000", 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036004779999999998 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.504697+0000", 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3001e-05 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:340 3.1s0 3:8a735ecd:::smithi12138290-30:head [setxattr _header (57) in=7b] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.497764+0000", 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.23743850599999999, 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0071883370000000004, 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.841 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 340 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497764+0000", 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497764+0000", 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497764+0000", 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9200000000000002e-07 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497766+0000", 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6819999999999999e-06 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497770+0000", 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4950000000000002e-06 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497773+0000", 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8420000000000001e-06 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.499959+0000", 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021855170000000001 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500054+0000", 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5581000000000005e-05 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500454+0000", 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00040002400000000002 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:31.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.502271+0000", 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018171140000000001 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.504906+0000", 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002634207 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.504952+0000", 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6582999999999998e-05 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:341 3.1s0 3:8a735ecd:::smithi12138290-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.23742565800000001, 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.008644522, 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:31.847 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 341 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:31.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4400000000000001e-07 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497779+0000", 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7260000000000001e-06 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:31.849 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497783+0000", 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9509999999999999e-06 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497786+0000", 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8629999999999999e-06 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500932+0000", 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031462389999999999 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.501005+0000", 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3546000000000004e-05 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.505207+0000", 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042014330000000001 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.505269+0000", 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2337999999999998e-05 2024-09-16T03:40:31.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:31.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:31.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:31.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506421+0000", 2024-09-16T03:40:31.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001152082 2024-09-16T03:40:31.852 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.852 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.852 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.852 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:31.852 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:31.852 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:31.854 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:40:31.854 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:31.851+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:31.955 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:31.953+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:31.958 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:31.958 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:31.958 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:31.958 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:31.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:40:32.056 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:32.054+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:32.069 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:32.069 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:32.069 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:32.069 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:32.069 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:32.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:40:32.171 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:32.155+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:32.223 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:32.223 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:32.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:32.223 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:32.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.223 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:213 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~688128] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:32.223 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.552584+0000", 2024-09-16T03:40:32.223 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.667713103000001, 2024-09-16T03:40:32.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059945609999999998, 2024-09-16T03:40:32.223 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 213 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.552584+0000", 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.552584+0000", 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.552587+0000", 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7300000000000001e-06 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.225 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.553514+0000", 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00092716800000000002 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.553527+0000", 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2991e-05 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.553533+0000", 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7350000000000001e-06 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.226 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.553604+0000", 2024-09-16T03:40:32.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1439999999999994e-05 2024-09-16T03:40:32.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.553667+0000", 2024-09-16T03:40:32.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2627999999999997e-05 2024-09-16T03:40:32.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.554786+0000", 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011192890000000001 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558234+0000", 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034485599999999998 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558488+0000", 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000253673 2024-09-16T03:40:32.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558578+0000", 2024-09-16T03:40:32.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0346999999999994e-05 2024-09-16T03:40:32.229 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.229 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.229 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.229 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:214 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.554734+0000", 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.66556344, 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0056718630000000001, 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 214 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.230 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.554734+0000", 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.554734+0000", 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.554735+0000", 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9900000000000009e-07 2024-09-16T03:40:32.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.232 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.232 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555372+0000", 2024-09-16T03:40:32.232 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00063754000000000003 2024-09-16T03:40:32.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555387+0000", 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4932e-05 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555405+0000", 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8292999999999999e-05 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555933+0000", 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00052810699999999999 2024-09-16T03:40:32.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556036+0000", 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102168 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.556829+0000", 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00079313699999999997 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558236+0000", 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014069169999999999 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.560361+0000", 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021251120000000002 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.560405+0000", 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4657999999999998e-05 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.235 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:215 3.bs0 3:de0fee80:::smithi12138290-48:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.555429+0000", 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.664868082, 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005874036, 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.236 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 215 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555429+0000", 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555429+0000", 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555430+0000", 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9199999999999999e-07 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555434+0000", 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6029999999999998e-06 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555443+0000", 2024-09-16T03:40:32.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8789999999999992e-06 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555450+0000", 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7109999999999998e-06 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557393+0000", 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019429829999999999 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557510+0000", 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011719599999999999 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558006+0000", 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00049595900000000003 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.560272+0000", 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022660139999999998 2024-09-16T03:40:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561233+0000", 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00096107099999999997 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561303+0000", 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9727999999999994e-05 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.242 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:216 3.bs0 3:de0fee80:::smithi12138290-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:32.242 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.555462+0000", 2024-09-16T03:40:32.242 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.664835262, 2024-09-16T03:40:32.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0071725290000000004, 2024-09-16T03:40:32.242 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.242 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.242 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:32.242 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.242 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 216 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555462+0000", 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555462+0000", 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555462+0000", 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555465+0000", 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555473+0000", 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.909e-06 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555479+0000", 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8950000000000003e-06 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558776+0000", 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032970579999999998 2024-09-16T03:40:32.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558831+0000", 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5133999999999998e-05 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561771+0000", 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029393380000000001 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561833+0000", 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2637000000000004e-05 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562634+0000", 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080094400000000005 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:273 3.bs0 3:db21b75b:::smithi12138290-3:head [read 0~49152,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.627761+0000", 2024-09-16T03:40:32.247 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 10.592536317, 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040900650000000004, 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 273 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.248 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627761+0000", 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627761+0000", 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627761+0000", 2024-09-16T03:40:32.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0100000000000004e-07 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627767+0000", 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5620000000000003e-06 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627779+0000", 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2125e-05 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.250 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627786+0000", 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9319999999999999e-06 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.629944+0000", 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021580549999999999 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630057+0000", 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000112681 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.631851+0000", 2024-09-16T03:40:32.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017940090000000001 2024-09-16T03:40:32.252 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.252 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.252 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:282 3.4s0 3:2307c0f5:::smithi12138290-28:head [append 0~458752] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.6455062920000003, 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0093867299999999994, 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 282 2024-09-16T03:40:32.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574793+0000", 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5799999999999999e-06 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575536+0000", 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00074309099999999998 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575567+0000", 2024-09-16T03:40:32.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0382000000000002e-05 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575590+0000", 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3215000000000001e-05 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575665+0000", 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5462000000000004e-05 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:32.256 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575671+0000", 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6969999999999998e-06 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577693+0000", 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020222439999999999 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577796+0000", 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010306000000000001 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.257 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578998+0000", 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012013880000000001 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.583553+0000", 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045549340000000001 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584072+0000", 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00051922199999999996 2024-09-16T03:40:32.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584177+0000", 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105455 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:283 3.4s0 3:2307c0f5:::smithi12138290-28:head [append 0~475136] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.577768+0000", 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.6425286420000003, 2024-09-16T03:40:32.259 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066247500000000004, 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 283 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577768+0000", 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577768+0000", 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577770+0000", 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.156e-06 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578319+0000", 2024-09-16T03:40:32.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00054978899999999996 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578336+0000", 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7014e-05 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578346+0000", 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4590000000000004e-06 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.262 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.263 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580005+0000", 2024-09-16T03:40:32.263 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00165913 2024-09-16T03:40:32.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.263 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.263 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.263 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580125+0000", 2024-09-16T03:40:32.263 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000120006 2024-09-16T03:40:32.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.263 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.263 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580986+0000", 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00086071900000000002 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.583554+0000", 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025686149999999998 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584323+0000", 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076837200000000002 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584393+0000", 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0489999999999998e-05 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:284 3.4s0 3:2307c0f5:::smithi12138290-28:head [append 0~294912] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.579743+0000", 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.6405538859999993, 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057780169999999999, 2024-09-16T03:40:32.265 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 284 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.579743+0000", 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.266 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.579743+0000", 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.579745+0000", 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8199999999999999e-06 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580117+0000", 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037198899999999998 2024-09-16T03:40:32.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580127+0000", 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0373e-05 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580133+0000", 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0449999999999997e-06 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.268 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.581790+0000", 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001656664 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.581988+0000", 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00019825499999999999 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.582917+0000", 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00092876799999999995 2024-09-16T03:40:32.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.583930+0000", 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010133539999999999 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.585449+0000", 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015182360000000001 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.585521+0000", 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2513000000000002e-05 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:285 3.4s0 3:2307c0f5:::smithi12138290-28:head [setxattr _header (56) in=7b] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.580152+0000", 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.6401446810000007, 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006825065, 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 285 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580152+0000", 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.273 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580152+0000", 2024-09-16T03:40:32.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.273 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.273 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580153+0000", 2024-09-16T03:40:32.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:40:32.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580159+0000", 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5280000000000003e-06 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580167+0000", 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3930000000000008e-06 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580173+0000", 2024-09-16T03:40:32.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8969999999999996e-06 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584551+0000", 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043776290000000001 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584616+0000", 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5457999999999996e-05 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.275 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584936+0000", 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00032023199999999999 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.586559+0000", 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001622559 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.586922+0000", 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00036351599999999998 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.586977+0000", 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5016999999999999e-05 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:286 3.4s0 3:2307c0f5:::smithi12138290-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.580183+0000", 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.6401144619999997, 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079802169999999995, 2024-09-16T03:40:32.277 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 286 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580183+0000", 2024-09-16T03:40:32.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580183+0000", 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580183+0000", 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4199999999999995e-07 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580186+0000", 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4430000000000002e-06 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580193+0000", 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0269999999999998e-06 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580198+0000", 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.079e-06 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.585225+0000", 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050276269999999998 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.585280+0000", 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4589999999999997e-05 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.587268+0000", 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019877639999999999 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.587356+0000", 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8437999999999996e-05 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.588163+0000", 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080660699999999998 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.282 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:311 3.bs0 3:db21b75b:::smithi12138290-3:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.72935419, 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013432349, 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.283 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 311 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:40:32.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490945+0000", 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.419e-06 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490953+0000", 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7379999999999994e-06 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490973+0000", 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0460999999999999e-05 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490984+0000", 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0857e-05 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491053+0000", 2024-09-16T03:40:32.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8351999999999999e-05 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491061+0000", 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3569999999999998e-06 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493588+0000", 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025267430000000001 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493683+0000", 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4876000000000004e-05 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496393+0000", 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027106500000000002 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504203+0000", 2024-09-16T03:40:32.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078100080000000002 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504276+0000", 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2601000000000005e-05 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504375+0000", 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9287e-05 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.289 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:312 3.bs0 3:db21b75b:::smithi12138290-3:head [rollback a] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.7292885650000001, 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013925697000000001, 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.290 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 312 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491009+0000", 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0200000000000001e-07 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491013+0000", 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8160000000000004e-06 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491022+0000", 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6649999999999992e-06 2024-09-16T03:40:32.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491028+0000", 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4760000000000003e-06 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491124+0000", 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6192000000000003e-05 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491132+0000", 2024-09-16T03:40:32.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6539999999999999e-06 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499113+0000", 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079811880000000002 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499184+0000", 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0749999999999999e-05 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499557+0000", 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037325200000000002 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504380+0000", 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048224590000000003 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504852+0000", 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047194899999999999 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504934+0000", 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2594000000000002e-05 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:313 3.bs0 3:db21b75b:::smithi12138290-3:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.7292469239999999, 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016369406999999999, 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.296 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 313 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491051+0000", 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.82e-07 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491054+0000", 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5260000000000002e-06 2024-09-16T03:40:32.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491063+0000", 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.354e-06 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491069+0000", 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2299999999999996e-06 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491177+0000", 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000108285 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491184+0000", 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0489999999999998e-06 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501958+0000", 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010773448 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502064+0000", 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106661 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502545+0000", 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000481077 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504381+0000", 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018352970000000001 2024-09-16T03:40:32.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.507349+0000", 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002968666 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.507420+0000", 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0232000000000001e-05 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s4 49/31 ECSubWrite(tid=30, reqid=client.4615.0:314, at_version=49'20, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.493704+0000", 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.7265932159999999, 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050398409999999998, 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493704+0000", 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493704+0000", 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493704+0000", 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493746+0000", 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2131e-05 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493758+0000", 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1056999999999999e-05 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493766+0000", 2024-09-16T03:40:32.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3820000000000004e-06 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493822+0000", 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5766000000000001e-05 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493831+0000", 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4040000000000001e-06 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.306 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.498718+0000", 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048865230000000003 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.498744+0000", 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6038e-05 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.307 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(osd.6.22:3 3.4s0 3:2ef36fe6:::smithi12138290-31:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.493828+0000", 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.7264688169999998, 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054416689999999997, 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493828+0000", 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493828+0000", 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493829+0000", 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.277e-06 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493836+0000", 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2190000000000001e-06 2024-09-16T03:40:32.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493848+0000", 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2434e-05 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493861+0000", 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3053e-05 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493942+0000", 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1291999999999993e-05 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494036+0000", 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3504999999999993e-05 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499270+0000", 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052338890000000002 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.311 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s4 49/31 ECSubWrite(tid=31, reqid=osd.8.0:31, at_version=49'22, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.495319+0000", 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.724978004, 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061288669999999996, 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495319+0000", 2024-09-16T03:40:32.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495319+0000", 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495320+0000", 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5700000000000002e-07 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495337+0000", 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7320999999999999e-05 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495348+0000", 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0804e-05 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495353+0000", 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7610000000000004e-06 2024-09-16T03:40:32.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495801+0000", 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044771900000000001 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495811+0000", 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4390000000000006e-06 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501409+0000", 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055981310000000001 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501448+0000", 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9134999999999997e-05 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 49/31 ECSubWriteReply(tid=40, last_complete=49'16, committed=1, applied=1))", 2024-09-16T03:40:32.316 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.497353+0000", 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.722943849, 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042628229999999998, 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497353+0000", 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497353+0000", 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497354+0000", 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.06e-07 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497358+0000", 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6829999999999999e-06 2024-09-16T03:40:32.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497370+0000", 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1674999999999999e-05 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497376+0000", 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7509999999999997e-06 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501597+0000", 2024-09-16T03:40:32.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042213779999999996 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501616+0000", 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8830000000000001e-05 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 49/31 ECSubWriteReply(tid=40, last_complete=49'16, committed=1, applied=1))", 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.497660+0000", 2024-09-16T03:40:32.320 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.7226368359999999, 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040951920000000001, 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497660+0000", 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497660+0000", 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497661+0000", 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6599999999999996e-07 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497665+0000", 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497680+0000", 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4581e-05 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497686+0000", 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6549999999999998e-06 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501725+0000", 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040389220000000003 2024-09-16T03:40:32.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501755+0000", 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0255000000000001e-05 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 49/31 ECSubWriteReply(tid=40, last_complete=49'16, committed=1, applied=1))", 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.497695+0000", 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.722602196, 2024-09-16T03:40:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040070119999999999, 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497695+0000", 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497695+0000", 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497696+0000", 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.387e-06 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497702+0000", 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0209999999999996e-06 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497716+0000", 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3682e-05 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497723+0000", 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5270000000000001e-06 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501687+0000", 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039632039999999997 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501702+0000", 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5191000000000001e-05 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s5 52/31 ECSubWrite(tid=37, reqid=client.4615.0:343, at_version=52'17, trim_to=0'0, roll_forward_to=42'14))", 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.502818+0000", 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.71747951700000001, 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004140754, 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.328 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.502818+0000", 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.502818+0000", 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.502818+0000", 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2600000000000001e-07 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.502839+0000", 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1077000000000002e-05 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.502857+0000", 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7931000000000002e-05 2024-09-16T03:40:32.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.502866+0000", 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9789999999999999e-06 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.502957+0000", 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0879999999999997e-05 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.502968+0000", 2024-09-16T03:40:32.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.075e-05 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506920+0000", 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039522209999999997 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506958+0000", 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8090000000000003e-05 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.333 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.333 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.333 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:32.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:40:32.335 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:32.332+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:32.434 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:32.432+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:32.478 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:32.478 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:32.478 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:32.478 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:32.486 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:40:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T03:40:32.499 INFO:tasks.rados.rados.0.smithi121.stdout:135: done (3 left) 2024-09-16T03:40:32.499 INFO:tasks.rados.rados.0.smithi121.stdout:136: rollback oid 19 current snap is 12 2024-09-16T03:40:32.499 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 19 to 1 2024-09-16T03:40:32.499 INFO:tasks.rados.rados.0.smithi121.stdout:132: finishing write tid 1 to smithi12138290-30 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:132: finishing write tid 2 to smithi12138290-30 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:132: finishing write tid 3 to smithi12138290-30 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:132: finishing write tid 4 to smithi12138290-30 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:133: finishing rollback tid 0 to smithi12138290-12 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:132: finishing write tid 5 to smithi12138290-30 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:132: oid 30 updating version 0 to 36 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:132: oid 30 updating version 36 to 39 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:132: oid 30 version 39 is already newer than 37 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:132: oid 30 version 39 is already newer than 38 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 30 v 39 (ObjNum 81 snap 12 seq_num 81) dirty exists 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:132: left oid 30 (ObjNum 81 snap 12 seq_num 81) 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:134: finishing copy_from racing read to smithi12138290-24 2024-09-16T03:40:32.500 INFO:tasks.rados.rados.0.smithi121.stdout:133: finishing rollback tid 1 to smithi12138290-12 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:133: finishing rollback tid 2 to smithi12138290-12 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 18 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:134: finishing copy_from to smithi12138290-24 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 24 v 4 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:132: done (3 left) 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:133: done (2 left) 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:134: done (1 left) 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:137: read oid 22 snap 1 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:137: expect deleted 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:138: rollback oid 29 current snap is 12 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 29 to 9 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 13 current snap is 12 2024-09-16T03:40:32.501 INFO:tasks.rados.rados.0.smithi121.stdout:139: seq_num 82 ranges {212992=622592,835584=475136,1310720=294912} 2024-09-16T03:40:32.505 INFO:tasks.rados.rados.0.smithi121.stdout:139: writing smithi12138290-13 from 212992 to 835584 tid 1 2024-09-16T03:40:32.507 INFO:tasks.rados.rados.0.smithi121.stdout:139: writing smithi12138290-13 from 835584 to 1310720 tid 2 2024-09-16T03:40:32.509 INFO:tasks.rados.rados.0.smithi121.stdout:139: writing smithi12138290-13 from 1310720 to 1605632 tid 3 2024-09-16T03:40:32.509 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 20 current snap is 12 2024-09-16T03:40:32.510 INFO:tasks.rados.rados.0.smithi121.stdout:140: seq_num 83 ranges {0=638976,638976=622592,1261568=147456} 2024-09-16T03:40:32.513 INFO:tasks.rados.rados.0.smithi121.stdout:140: writing smithi12138290-20 from 0 to 638976 tid 1 2024-09-16T03:40:32.518 INFO:tasks.rados.rados.0.smithi121.stdout:140: writing smithi12138290-20 from 638976 to 1261568 tid 2 2024-09-16T03:40:32.519 INFO:tasks.rados.rados.0.smithi121.stdout:140: writing smithi12138290-20 from 1261568 to 1409024 tid 3 2024-09-16T03:40:32.519 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 31 current snap is 12 2024-09-16T03:40:32.519 INFO:tasks.rados.rados.0.smithi121.stdout:141: seq_num 84 ranges {507904=655360,1163264=638976,1802240=131072} 2024-09-16T03:40:32.525 INFO:tasks.rados.rados.0.smithi121.stdout:141: writing smithi12138290-31 from 507904 to 1163264 tid 1 2024-09-16T03:40:32.529 INFO:tasks.rados.rados.0.smithi121.stdout:141: writing smithi12138290-31 from 1163264 to 1802240 tid 2 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:141: writing smithi12138290-31 from 1802240 to 1933312 tid 3 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:142: rollback oid 12 current snap is 12 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 12 to 9 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:136: finishing rollback tid 0 to smithi12138290-19 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:136: finishing rollback tid 1 to smithi12138290-19 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:136: finishing rollback tid 2 to smithi12138290-19 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 19 v 39 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:138: finishing rollback tid 0 to smithi12138290-29 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:138: finishing rollback tid 1 to smithi12138290-29 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:138: finishing rollback tid 2 to smithi12138290-29 2024-09-16T03:40:32.531 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 29 v 10 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:40:32.532 INFO:tasks.rados.rados.0.smithi121.stdout:139: finishing write tid 1 to smithi12138290-13 2024-09-16T03:40:32.532 INFO:tasks.rados.rados.0.smithi121.stdout:136: done (6 left) 2024-09-16T03:40:32.532 INFO:tasks.rados.rados.0.smithi121.stdout:137: done (5 left) 2024-09-16T03:40:32.532 INFO:tasks.rados.rados.0.smithi121.stdout:138: done (4 left) 2024-09-16T03:40:32.532 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 40 current snap is 12 2024-09-16T03:40:32.532 INFO:tasks.rados.rados.0.smithi121.stdout:139: finishing write tid 2 to smithi12138290-13 2024-09-16T03:40:32.532 INFO:tasks.rados.rados.0.smithi121.stdout:143: seq_num 85 ranges {671744=671744,1343488=770048} 2024-09-16T03:40:32.535 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:32.534+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:32.537 INFO:tasks.rados.rados.0.smithi121.stdout:143: writing smithi12138290-40 from 671744 to 1343488 tid 1 2024-09-16T03:40:32.544 INFO:tasks.rados.rados.0.smithi121.stdout:143: writing smithi12138290-40 from 1343488 to 2113536 tid 2 2024-09-16T03:40:32.544 INFO:tasks.rados.rados.0.smithi121.stdout:144: rollback oid 48 current snap is 12 2024-09-16T03:40:32.544 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 48 to 1 2024-09-16T03:40:32.544 INFO:tasks.rados.rados.0.smithi121.stdout:139: finishing write tid 3 to smithi12138290-13 2024-09-16T03:40:32.544 INFO:tasks.rados.rados.0.smithi121.stdout:139: finishing write tid 4 to smithi12138290-13 2024-09-16T03:40:32.544 INFO:tasks.rados.rados.0.smithi121.stdout:140: finishing write tid 1 to smithi12138290-20 2024-09-16T03:40:32.544 INFO:tasks.rados.rados.0.smithi121.stdout:139: finishing write tid 5 to smithi12138290-13 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:139: oid 13 updating version 0 to 14 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:139: oid 13 updating version 14 to 15 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:139: oid 13 version 15 is already newer than 13 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:139: oid 13 version 15 is already newer than 12 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 13 v 15 (ObjNum 82 snap 12 seq_num 82) dirty exists 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:139: left oid 13 (ObjNum 82 snap 12 seq_num 82) 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:140: finishing write tid 2 to smithi12138290-20 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:140: finishing write tid 3 to smithi12138290-20 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:140: finishing write tid 4 to smithi12138290-20 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:140: finishing write tid 5 to smithi12138290-20 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:140: oid 20 updating version 0 to 21 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:140: oid 20 updating version 21 to 22 2024-09-16T03:40:32.545 INFO:tasks.rados.rados.0.smithi121.stdout:140: oid 20 updating version 22 to 24 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:140: oid 20 version 24 is already newer than 23 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 20 v 24 (ObjNum 83 snap 12 seq_num 83) dirty exists 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:140: left oid 20 (ObjNum 83 snap 12 seq_num 83) 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:141: finishing write tid 1 to smithi12138290-31 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:139: done (5 left) 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:140: done (4 left) 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:145: read oid 24 snap 9 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:145: expect (ObjNum 72 snap 6 seq_num 72) 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:146: read oid 35 snap 11 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:146: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 26 current snap is 12 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:142: finishing rollback tid 0 to smithi12138290-12 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:141: finishing write tid 2 to smithi12138290-31 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:142: finishing rollback tid 1 to smithi12138290-12 2024-09-16T03:40:32.546 INFO:tasks.rados.rados.0.smithi121.stdout:142: finishing rollback tid 2 to smithi12138290-12 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 27 (ObjNum 70 snap 5 seq_num 70) dirty exists 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:141: finishing write tid 3 to smithi12138290-31 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:141: finishing write tid 4 to smithi12138290-31 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:141: finishing write tid 5 to smithi12138290-31 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:141: oid 31 updating version 0 to 15 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:141: oid 31 version 15 is already newer than 13 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:141: oid 31 version 15 is already newer than 14 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:141: oid 31 updating version 15 to 16 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 31 v 16 (ObjNum 84 snap 12 seq_num 84) dirty exists 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:141: left oid 31 (ObjNum 84 snap 12 seq_num 84) 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:147: seq_num 86 ranges {0=688128,688128=540672,1228800=229376} 2024-09-16T03:40:32.547 INFO:tasks.rados.rados.0.smithi121.stdout:147: writing smithi12138290-26 from 0 to 688128 tid 1 2024-09-16T03:40:32.551 INFO:tasks.rados.rados.0.smithi121.stdout:147: writing smithi12138290-26 from 688128 to 1228800 tid 2 2024-09-16T03:40:32.553 INFO:tasks.rados.rados.0.smithi121.stdout:147: writing smithi12138290-26 from 1228800 to 1458176 tid 3 2024-09-16T03:40:32.553 INFO:tasks.rados.rados.0.smithi121.stdout:141: done (6 left) 2024-09-16T03:40:32.553 INFO:tasks.rados.rados.0.smithi121.stdout:142: done (5 left) 2024-09-16T03:40:32.553 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 46 current snap is 12 2024-09-16T03:40:32.553 INFO:tasks.rados.rados.0.smithi121.stdout:143: finishing write tid 1 to smithi12138290-40 2024-09-16T03:40:32.553 INFO:tasks.rados.rados.0.smithi121.stdout:148: seq_num 87 ranges {770048=704512,1474560=688128,2162688=81920} 2024-09-16T03:40:32.561 INFO:tasks.rados.rados.0.smithi121.stdout:148: writing smithi12138290-46 from 770048 to 1474560 tid 1 2024-09-16T03:40:32.565 INFO:tasks.rados.rados.0.smithi121.stdout:148: writing smithi12138290-46 from 1474560 to 2162688 tid 2 2024-09-16T03:40:32.566 INFO:tasks.rados.rados.0.smithi121.stdout:148: writing smithi12138290-46 from 2162688 to 2244608 tid 3 2024-09-16T03:40:32.566 INFO:tasks.rados.rados.0.smithi121.stdout:149: read oid 28 snap -1 2024-09-16T03:40:32.566 INFO:tasks.rados.rados.0.smithi121.stdout:149: expect (ObjNum 73 snap 6 seq_num 73) 2024-09-16T03:40:32.566 INFO:tasks.rados.rados.0.smithi121.stdout:150: read oid 45 snap -1 2024-09-16T03:40:32.566 INFO:tasks.rados.rados.0.smithi121.stdout:150: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-16T03:40:32.567 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 50 current snap is 12 2024-09-16T03:40:32.567 INFO:tasks.rados.rados.0.smithi121.stdout:151: seq_num 88 ranges {835584=720896,1556480=442368,1998848=327680} 2024-09-16T03:40:32.576 INFO:tasks.rados.rados.0.smithi121.stdout:151: writing smithi12138290-50 from 835584 to 1556480 tid 1 2024-09-16T03:40:32.581 INFO:tasks.rados.rados.0.smithi121.stdout:151: writing smithi12138290-50 from 1556480 to 1998848 tid 2 2024-09-16T03:40:32.581 INFO:tasks.rados.rados.0.smithi121.stdout:151: writing smithi12138290-50 from 1998848 to 2326528 tid 3 2024-09-16T03:40:32.581 INFO:tasks.rados.rados.0.smithi121.stdout:152: read oid 44 snap -1 2024-09-16T03:40:32.581 INFO:tasks.rados.rados.0.smithi121.stdout:152: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-16T03:40:32.581 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 18 current snap is 12 2024-09-16T03:40:32.581 INFO:tasks.rados.rados.0.smithi121.stdout:144: finishing rollback tid 0 to smithi12138290-48 2024-09-16T03:40:32.581 INFO:tasks.rados.rados.0.smithi121.stdout:144: finishing rollback tid 1 to smithi12138290-48 2024-09-16T03:40:32.581 INFO:tasks.rados.rados.0.smithi121.stdout:146: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:40:32.596 INFO:tasks.rados.rados.0.smithi121.stdout:143: finishing write tid 2 to smithi12138290-40 2024-09-16T03:40:32.597 INFO:tasks.rados.rados.0.smithi121.stdout:144: finishing rollback tid 2 to smithi12138290-48 2024-09-16T03:40:32.597 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 48 v 28 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:40:32.597 INFO:tasks.rados.rados.0.smithi121.stdout:143: finishing write tid 3 to smithi12138290-40 2024-09-16T03:40:32.597 INFO:tasks.rados.rados.0.smithi121.stdout:153: seq_num 89 ranges {294912=737280,1032192=573440,1605632=196608} 2024-09-16T03:40:32.602 INFO:tasks.rados.rados.0.smithi121.stdout:153: writing smithi12138290-18 from 294912 to 1032192 tid 1 2024-09-16T03:40:32.605 INFO:tasks.rados.rados.0.smithi121.stdout:153: writing smithi12138290-18 from 1032192 to 1605632 tid 2 2024-09-16T03:40:32.607 INFO:tasks.rados.rados.0.smithi121.stdout:153: writing smithi12138290-18 from 1605632 to 1802240 tid 3 2024-09-16T03:40:32.607 INFO:tasks.rados.rados.0.smithi121.stdout:144: done (10 left) 2024-09-16T03:40:32.607 INFO:tasks.rados.rados.0.smithi121.stdout:146: done (9 left) 2024-09-16T03:40:32.607 INFO:tasks.rados.rados.0.smithi121.stdout:154: snap_remove snap 11 2024-09-16T03:40:32.613 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:32.613 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:32.613 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:32.613 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:32.613 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:32.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:40:32.665 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:32.664+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:32.691 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:40:32.691 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:40:32.691 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-09-16T03:40:32.691 INFO:tasks.thrashosds.thrasher:Killing osd 13, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:40:32.691 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-16T03:40:32.691 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:40:32.717 INFO:tasks.ceph.osd.13:Stopped 2024-09-16T03:40:32.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 13 2024-09-16T03:40:32.728 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:32.728 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:32.728 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:32.728 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:32.728 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.728 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:205 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~655360] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:32.728 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:40:32.728 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 16.202481533, 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075538640000000004, 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 205 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.729 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523025+0000", 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3519999999999999e-06 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.730 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523819+0000", 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00079422800000000001 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523834+0000", 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5333e-05 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523842+0000", 2024-09-16T03:40:32.731 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2960000000000001e-06 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523903+0000", 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1124000000000006e-05 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523981+0000", 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8166999999999999e-05 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525035+0000", 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010540969999999999 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528679+0000", 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036443090000000001 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530521+0000", 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018416789999999999 2024-09-16T03:40:32.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530577+0000", 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6279000000000001e-05 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:206 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.525364+0000", 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 16.200140698999999, 2024-09-16T03:40:32.734 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055004260000000001, 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 206 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525364+0000", 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.735 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525364+0000", 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525365+0000", 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1209999999999999e-06 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.736 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525823+0000", 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000457641 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525836+0000", 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2850999999999999e-05 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525842+0000", 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.737 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526108+0000", 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026636800000000001 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526224+0000", 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116209 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527074+0000", 2024-09-16T03:40:32.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00084922299999999997 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528682+0000", 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0016087779999999999 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530821+0000", 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00213875 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530865+0000", 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3625000000000003e-05 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:207 3.6s0 3:656a4357:::smithi12138290-23:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.525856+0000", 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 16.199648789000001, 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054829939999999997, 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.740 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 207 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525856+0000", 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525856+0000", 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525857+0000", 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7000000000000004e-07 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525859+0000", 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4380000000000002e-06 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525865+0000", 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6919999999999999e-06 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525870+0000", 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1889999999999998e-06 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527700+0000", 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001829682 2024-09-16T03:40:32.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527793+0000", 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2841000000000001e-05 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528259+0000", 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00046626300000000002 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528747+0000", 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00048799999999999999 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.531285+0000", 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025385569999999999 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.531339+0000", 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3662000000000002e-05 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.745 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:208 3.6s0 3:656a4357:::smithi12138290-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.525875+0000", 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 16.199629449, 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006650698, 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.746 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 208 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525875+0000", 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525875+0000", 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525876+0000", 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525877+0000", 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.477e-06 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525883+0000", 2024-09-16T03:40:32.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2689999999999999e-06 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525895+0000", 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2384e-05 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528574+0000", 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002678854 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:32.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528669+0000", 2024-09-16T03:40:32.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4968000000000003e-05 2024-09-16T03:40:32.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.531658+0000", 2024-09-16T03:40:32.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029888029999999999 2024-09-16T03:40:32.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.531711+0000", 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3180999999999998e-05 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.532526+0000", 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00081546099999999996 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(osd.1.22:1 3.6s0 3:616ac6f8:::smithi12138290-8:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:32.751 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.558744+0000", 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 11.166761106999999, 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041018460000000001, 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558744+0000", 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558744+0000", 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558745+0000", 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.635e-06 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558751+0000", 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8610000000000003e-06 2024-09-16T03:40:32.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558760+0000", 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6430000000000001e-06 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558768+0000", 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6490000000000008e-06 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558825+0000", 2024-09-16T03:40:32.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7796e-05 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558951+0000", 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012591100000000001 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.562846+0000", 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038943509999999999 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:305 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.238255799, 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087742589999999995, 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 305 2024-09-16T03:40:32.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487251+0000", 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.824e-06 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487259+0000", 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1689999999999992e-06 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487275+0000", 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5889e-05 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487283+0000", 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9710000000000004e-06 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487385+0000", 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010225099999999999 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487554+0000", 2024-09-16T03:40:32.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016910199999999999 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489000+0000", 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001445499 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492361+0000", 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033615799999999999 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495935+0000", 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0035732899999999998 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496023+0000", 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8684000000000007e-05 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:306 3.6s0 3:64535dd7:::smithi12138290-18:head [rollback 5] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:32.761 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.2382029369999996, 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0099232220000000006, 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 306 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487303+0000", 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.44e-07 2024-09-16T03:40:32.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487306+0000", 2024-09-16T03:40:32.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.817e-06 2024-09-16T03:40:32.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487314+0000", 2024-09-16T03:40:32.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4000000000000003e-06 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487319+0000", 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2379999999999997e-06 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489535+0000", 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002216347 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489617+0000", 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1559999999999998e-05 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490002+0000", 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038484000000000001 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492363+0000", 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023609070000000002 2024-09-16T03:40:32.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497154+0000", 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047910110000000004 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497225+0000", 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1557999999999995e-05 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:307 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.2381611210000001, 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010112101, 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 307 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.768 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.51e-07 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487356+0000", 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1793999999999999e-05 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487362+0000", 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8100000000000003e-06 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487366+0000", 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8670000000000004e-06 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490302+0000", 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029359299999999998 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490396+0000", 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4134000000000004e-05 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490929+0000", 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00053311500000000002 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492364+0000", 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014347279999999999 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497383+0000", 2024-09-16T03:40:32.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050189220000000003 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497456+0000", 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3250000000000005e-05 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(osd.0.23:4 3.6s0 3:616ac6f8:::smithi12138290-8:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:32.773 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.500407+0000", 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.225097793, 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003807871, 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500407+0000", 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500407+0000", 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500409+0000", 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6980000000000001e-06 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500415+0000", 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6899999999999997e-06 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500425+0000", 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0685e-05 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500433+0000", 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7190000000000001e-06 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500495+0000", 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1826000000000001e-05 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500504+0000", 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1800000000000002e-06 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.503070+0000", 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002566343 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.503161+0000", 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0532000000000007e-05 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.504215+0000", 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010541979999999999 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.778 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:348 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498255+0000", 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.22724961099999999, 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069791749999999998, 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 348 2024-09-16T03:40:32.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498255+0000", 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498255+0000", 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498257+0000", 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4649999999999999e-06 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498264+0000", 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5440000000000001e-06 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498276+0000", 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1324e-05 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498283+0000", 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5730000000000002e-06 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498455+0000", 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000171533 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498560+0000", 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105282 2024-09-16T03:40:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499260+0000", 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000700042 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505047+0000", 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057870489999999998 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505134+0000", 2024-09-16T03:40:32.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6939000000000002e-05 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505235+0000", 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000100424 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:349 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [rollback 3] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:32.784 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.22720658499999999, 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084181240000000008, 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 349 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498299+0000", 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6600000000000002e-07 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498301+0000", 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3860000000000001e-06 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498307+0000", 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8590000000000001e-06 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498312+0000", 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6249999999999998e-06 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499617+0000", 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0013049660000000001 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499672+0000", 2024-09-16T03:40:32.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5787e-05 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500118+0000", 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044587499999999998 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505378+0000", 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005259789 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506634+0000", 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012560169999999999 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506716+0000", 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2354000000000005e-05 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:350 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:32.790 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.22718651300000001, 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086865829999999995, 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 350 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498319+0000", 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.39e-07 2024-09-16T03:40:32.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498321+0000", 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.815e-06 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498325+0000", 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8749999999999999e-06 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498329+0000", 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.014e-06 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500398+0000", 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020690069999999999 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500458+0000", 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9429999999999999e-05 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500725+0000", 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026708899999999998 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505380+0000", 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046551040000000002 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506933+0000", 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015530769999999999 2024-09-16T03:40:32.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507005+0000", 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1833000000000002e-05 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s2 53/31 ECSubWrite(tid=40, reqid=osd.13.0:40, at_version=53'20, trim_to=0'0, roll_forward_to=52'18))", 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.22391918099999999, 2024-09-16T03:40:32.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007493184, 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.23e-07 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501604+0000", 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8465e-05 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501615+0000", 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.06e-05 2024-09-16T03:40:32.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501623+0000", 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8019999999999992e-06 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501779+0000", 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000156093 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501788+0000", 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3840000000000003e-06 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.509033+0000", 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0072446300000000002 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.509079+0000", 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5986999999999999e-05 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.800 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.bs6 53/31 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=53'24, trim_to=0'0, roll_forward_to=51'22))", 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.508853+0000", 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.21665193099999999, 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046456709999999997, 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508853+0000", 2024-09-16T03:40:32.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508853+0000", 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508853+0000", 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508875+0000", 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2140000000000001e-05 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508884+0000", 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6330000000000002e-06 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508897+0000", 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2807e-05 2024-09-16T03:40:32.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508958+0000", 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0810999999999998e-05 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508966+0000", 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6289999999999999e-06 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513452+0000", 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044858160000000001 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513499+0000", 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6554999999999997e-05 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s2 53/31 ECSubWrite(tid=42, reqid=client.4615.0:362, at_version=53'21, trim_to=0'0, roll_forward_to=53'20))", 2024-09-16T03:40:32.805 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.515864+0000", 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.20964079099999999, 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055959920000000002, 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.515864+0000", 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.515864+0000", 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.515865+0000", 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0900000000000001e-07 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516245+0000", 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038044000000000001 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516260+0000", 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4948e-05 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516269+0000", 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9430000000000006e-06 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516348+0000", 2024-09-16T03:40:32.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8817000000000001e-05 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516357+0000", 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3500000000000003e-06 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.521430+0000", 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050732219999999996 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.521460+0000", 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9663e-05 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:412 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~737280] snapc d=[d,b,8,5,3] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.122573774, 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007659028, 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.810 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 412 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602932+0000", 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3499999999999995e-07 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603668+0000", 2024-09-16T03:40:32.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073570899999999997 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603677+0000", 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4180000000000003e-06 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603702+0000", 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5225999999999999e-05 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603754+0000", 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1391000000000003e-05 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603826+0000", 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2648e-05 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604587+0000", 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076084500000000001 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.608450+0000", 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038632409999999999 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610532+0000", 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020811580000000001 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610590+0000", 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8656999999999998e-05 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.815 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:413 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~573440] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.604921+0000", 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.120583513, 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060231779999999997, 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.816 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 413 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604921+0000", 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604921+0000", 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604922+0000", 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0440000000000001e-06 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606010+0000", 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001087743 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606023+0000", 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2384e-05 2024-09-16T03:40:32.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606029+0000", 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3160000000000002e-06 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606071+0000", 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1727999999999998e-05 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606140+0000", 2024-09-16T03:40:32.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8895999999999999e-05 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606663+0000", 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00052336799999999997 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610599+0000", 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039365729999999996 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610895+0000", 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00029558699999999999 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610945+0000", 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9539000000000001e-05 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:414 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~196608] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.606301+0000", 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.119203456, 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055128699999999996, 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 414 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606301+0000", 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606301+0000", 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606302+0000", 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7000000000000004e-07 2024-09-16T03:40:32.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606513+0000", 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021062399999999999 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606520+0000", 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0040000000000002e-06 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606529+0000", 2024-09-16T03:40:32.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5750000000000007e-06 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.608634+0000", 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002105015 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.608742+0000", 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010727500000000001 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.609405+0000", 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00066365700000000001 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610601+0000", 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001195242 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.611743+0000", 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00114207 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.611814+0000", 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1737999999999997e-05 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:415 3.6s0 3:64535dd7:::smithi12138290-18:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.118967224, 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0056604400000000001, 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 415 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.53e-07 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606540+0000", 2024-09-16T03:40:32.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0430000000000002e-06 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606545+0000", 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5059999999999997e-06 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606547+0000", 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7530000000000002e-06 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.609850+0000", 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003302471 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.609914+0000", 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3992000000000007e-05 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610241+0000", 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000326767 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610789+0000", 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00054870099999999996 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612138+0000", 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001348807 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612198+0000", 2024-09-16T03:40:32.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0047000000000003e-05 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:416 3.6s0 3:64535dd7:::smithi12138290-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.606551+0000", 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.118953657, 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068823779999999998, 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:32.833 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 416 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606551+0000", 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606551+0000", 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:32.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606552+0000", 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5699999999999999e-07 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606553+0000", 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.297e-06 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606560+0000", 2024-09-16T03:40:32.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2919999999999998e-06 2024-09-16T03:40:32.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:32.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606563+0000", 2024-09-16T03:40:32.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0570000000000001e-06 2024-09-16T03:40:32.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.611182+0000", 2024-09-16T03:40:32.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046187349999999997 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.611234+0000", 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2080000000000003e-05 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612474+0000", 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001239892 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612554+0000", 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9999000000000004e-05 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.613434+0000", 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00087976899999999997 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:32.838 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:32.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:40:32.840 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:32.838+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:32.941 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:32.938+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:32.946 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:32.946 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:32.947 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:32.947 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:32.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:40:33.005 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T03:40:33.004+0000 7f011120c640 -1 mon.a@0(leader).osd e53 definitely_dead 0 2024-09-16T03:40:33.041 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:33.040+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:33.098 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:33.098 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:33.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:40:33.150 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:33.148+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:33.210 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:33.210 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 37/31 ECSubWrite(tid=23, reqid=client.4615.0:205, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 16.682748144000001, 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050337250000000002, 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:33.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9999999999999999e-07 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526147+0000", 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00092697700000000005 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526170+0000", 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3377e-05 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526182+0000", 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1331999999999999e-05 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526338+0000", 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000156009 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526347+0000", 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7610000000000002e-06 2024-09-16T03:40:33.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530193+0000", 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003845209 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530254+0000", 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0859999999999997e-05 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 42/31 ECSubWrite(tid=73, reqid=osd.0.0:73, at_version=42'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.555894+0000", 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.652073855999999, 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041093190000000002, 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555894+0000", 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555894+0000", 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555894+0000", 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7100000000000001e-07 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555910+0000", 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5661000000000001e-05 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555921+0000", 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1181e-05 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555933+0000", 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1949e-05 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555984+0000", 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0584000000000002e-05 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.555990+0000", 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5969999999999997e-06 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.559966+0000", 2024-09-16T03:40:33.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039761299999999996 2024-09-16T03:40:33.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.560003+0000", 2024-09-16T03:40:33.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:40:33.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.220 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 42/31 ECSubWrite(tid=30, reqid=osd.14.0:30, at_version=42'20, trim_to=0'0, roll_forward_to=41'18))", 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.557656+0000", 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.650312045, 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047645880000000002, 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557656+0000", 2024-09-16T03:40:33.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557656+0000", 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557657+0000", 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.68e-07 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557674+0000", 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7887e-05 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557696+0000", 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2035000000000002e-05 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557705+0000", 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5750000000000001e-06 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557774+0000", 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9015000000000002e-05 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557785+0000", 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1289e-05 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562388+0000", 2024-09-16T03:40:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004602783 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562420+0000", 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2335999999999997e-05 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 42/31 ECSubWrite(tid=74, reqid=osd.0.0:73, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.562839+0000", 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.645128455, 2024-09-16T03:40:33.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047921420000000001, 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562839+0000", 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562839+0000", 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562840+0000", 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9899999999999999e-07 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562852+0000", 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1975e-05 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562863+0000", 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1100999999999999e-05 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562869+0000", 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9139999999999996e-06 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562921+0000", 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1564999999999998e-05 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562927+0000", 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.455e-06 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.567594+0000", 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046666099999999999 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.567632+0000", 2024-09-16T03:40:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7722999999999998e-05 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 42/31 ECSubWrite(tid=32, reqid=osd.14.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.644242608000001, 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075866950000000001, 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563726+0000", 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.09e-07 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563741+0000", 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5123000000000001e-05 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563761+0000", 2024-09-16T03:40:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9335999999999999e-05 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563770+0000", 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3770000000000002e-06 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563840+0000", 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9725000000000001e-05 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563851+0000", 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1107000000000001e-05 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571277+0000", 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074264830000000002 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571312+0000", 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4635000000000003e-05 2024-09-16T03:40:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/31 ECSubWrite(tid=24, reqid=client.4615.0:268, at_version=42'15, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.629148+0000", 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.578819615, 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053360220000000002, 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629148+0000", 2024-09-16T03:40:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629148+0000", 2024-09-16T03:40:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629149+0000", 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8400000000000003e-07 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629190+0000", 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0806999999999997e-05 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629227+0000", 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6551000000000003e-05 2024-09-16T03:40:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629236+0000", 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7769999999999997e-06 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629292+0000", 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5980999999999998e-05 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629300+0000", 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4909999999999999e-06 2024-09-16T03:40:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634449+0000", 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051495170000000002 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634484+0000", 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5014000000000001e-05 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/31 ECSubWrite(tid=25, reqid=client.4615.0:269, at_version=42'16, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.577971225000001, 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004694803, 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4900000000000005e-07 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630013+0000", 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5722e-05 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630025+0000", 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2234e-05 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630033+0000", 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3560000000000004e-06 2024-09-16T03:40:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631638+0000", 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00160494 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631647+0000", 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4960000000000002e-06 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634669+0000", 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003022247 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634692+0000", 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2059e-05 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=30, reqid=client.4615.0:301, at_version=49'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7208535390000002, 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083935009999999994, 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:40:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487115+0000", 2024-09-16T03:40:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.36e-07 2024-09-16T03:40:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487566+0000", 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045117600000000002 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487581+0000", 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5092000000000001e-05 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487591+0000", 2024-09-16T03:40:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.058e-05 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487677+0000", 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5635000000000003e-05 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487685+0000", 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2830000000000001e-06 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495470+0000", 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077842739999999999 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495508+0000", 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8225000000000001e-05 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 49/31 ECSubWrite(tid=34, reqid=client.4615.0:305, at_version=49'22, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.489031+0000", 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7189365819999995, 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064300850000000003, 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489031+0000", 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489031+0000", 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489032+0000", 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4700000000000002e-07 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489060+0000", 2024-09-16T03:40:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000002e-05 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489076+0000", 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6013000000000001e-05 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489086+0000", 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9499999999999996e-06 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489306+0000", 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022061699999999999 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489317+0000", 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0895e-05 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495421+0000", 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061040540000000003 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495461+0000", 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0139000000000003e-05 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=31, reqid=osd.6.0:31, at_version=49'22, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.489235+0000", 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7187324769999996, 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0063054690000000002, 2024-09-16T03:40:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489235+0000", 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489235+0000", 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489236+0000", 2024-09-16T03:40:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5499999999999996e-07 2024-09-16T03:40:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489261+0000", 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5045000000000002e-05 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489275+0000", 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3743e-05 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489283+0000", 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2109999999999998e-06 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490215+0000", 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093184699999999997 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490227+0000", 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1737e-05 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495519+0000", 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052920049999999998 2024-09-16T03:40:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495541+0000", 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1926e-05 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 49/31 ECSubWrite(tid=35, reqid=client.4615.0:306, at_version=49'23, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7176511510000001, 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065860449999999996, 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490318+0000", 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5600000000000004e-07 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490342+0000", 2024-09-16T03:40:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4006e-05 2024-09-16T03:40:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490356+0000", 2024-09-16T03:40:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4449e-05 2024-09-16T03:40:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490366+0000", 2024-09-16T03:40:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8689999999999999e-06 2024-09-16T03:40:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494506+0000", 2024-09-16T03:40:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041400029999999997 2024-09-16T03:40:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494516+0000", 2024-09-16T03:40:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0229e-05 2024-09-16T03:40:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496868+0000", 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023520899999999998 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496903+0000", 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4443000000000002e-05 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=32, reqid=client.4615.0:302, at_version=49'23, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.491000+0000", 2024-09-16T03:40:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7169677029999999, 2024-09-16T03:40:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065612769999999999, 2024-09-16T03:40:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491000+0000", 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491000+0000", 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491001+0000", 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4600000000000003e-07 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492933+0000", 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019313749999999999 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492951+0000", 2024-09-16T03:40:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8521000000000001e-05 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492960+0000", 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4089999999999992e-06 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494238+0000", 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012774049999999999 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494247+0000", 2024-09-16T03:40:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0839999999999998e-06 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497522+0000", 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032754910000000002 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497562+0000", 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9045999999999999e-05 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 49/31 ECSubWrite(tid=36, reqid=client.4615.0:307, at_version=49'24, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.491223+0000", 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7167445260000003, 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057044000000000001, 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491223+0000", 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491223+0000", 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491224+0000", 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4399999999999998e-07 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491244+0000", 2024-09-16T03:40:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9286000000000001e-05 2024-09-16T03:40:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491259+0000", 2024-09-16T03:40:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5075999999999999e-05 2024-09-16T03:40:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491267+0000", 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3930000000000008e-06 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495031+0000", 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037640339999999999 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495041+0000", 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9969999999999993e-06 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496907+0000", 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018654380000000001 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496928+0000", 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1231999999999999e-05 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 49/31 ECSubWrite(tid=83, reqid=osd.0.0:83, at_version=49'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.491639+0000", 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7163292439999998, 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047024830000000004, 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491639+0000", 2024-09-16T03:40:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491639+0000", 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491639+0000", 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6800000000000001e-07 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491666+0000", 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6568e-05 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491682+0000", 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6280999999999999e-05 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491691+0000", 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3729999999999999e-06 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491771+0000", 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0195e-05 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491783+0000", 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1912e-05 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496306+0000", 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045223049999999999 2024-09-16T03:40:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496341+0000", 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5481000000000001e-05 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=33, reqid=client.4615.0:303, at_version=49'24, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.492978+0000", 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7149895180000003, 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046125319999999999, 2024-09-16T03:40:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492978+0000", 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492978+0000", 2024-09-16T03:40:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492979+0000", 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1600000000000001e-07 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492995+0000", 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5557999999999999e-05 2024-09-16T03:40:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.493005+0000", 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0686000000000001e-05 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.493012+0000", 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0169999999999999e-06 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495579+0000", 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002566381 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495588+0000", 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4399999999999994e-06 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497566+0000", 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019781640000000001 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497591+0000", 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4669999999999999e-05 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 49/31 ECSubWrite(tid=37, reqid=client.4615.0:307, at_version=0'0, trim_to=0'0, roll_forward_to=49'24))", 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.497641+0000", 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7103271580000001, 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004249231, 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497641+0000", 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497641+0000", 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497642+0000", 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.11e-06 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497654+0000", 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2023e-05 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497667+0000", 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3499999999999999e-05 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497677+0000", 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3859999999999996e-06 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497915+0000", 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023817200000000001 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497925+0000", 2024-09-16T03:40:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5699999999999999e-06 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.501850+0000", 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003925792 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.501890+0000", 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9678000000000002e-05 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=34, reqid=client.4615.0:303, at_version=0'0, trim_to=0'0, roll_forward_to=49'24))", 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.502228+0000", 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7057401030000001, 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005033989, 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502228+0000", 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502228+0000", 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502229+0000", 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5600000000000004e-07 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502240+0000", 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1653e-05 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502273+0000", 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.286e-05 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502283+0000", 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5789999999999993e-06 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502366+0000", 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3085999999999997e-05 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502380+0000", 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.412e-05 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.507229+0000", 2024-09-16T03:40:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048493770000000002 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.507262+0000", 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2458e-05 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 49/31 ECSubWrite(tid=32, reqid=osd.8.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=49'22))", 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.502355+0000", 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7056129740000001, 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048604859999999998, 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502355+0000", 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502355+0000", 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502356+0000", 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.95e-07 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502368+0000", 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1558e-05 2024-09-16T03:40:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502383+0000", 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5869999999999999e-05 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502392+0000", 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6680000000000007e-06 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502471+0000", 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8912999999999995e-05 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502482+0000", 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0567000000000001e-05 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.507178+0000", 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046967110000000001 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.507215+0000", 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7203999999999998e-05 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=35, reqid=client.4615.0:309, at_version=49'26, trim_to=0'0, roll_forward_to=49'24))", 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.504135+0000", 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7038332670000003, 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054625289999999998, 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504135+0000", 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504135+0000", 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504136+0000", 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6400000000000001e-07 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504450+0000", 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031444699999999998 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504462+0000", 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2208999999999999e-05 2024-09-16T03:40:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504471+0000", 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2129999999999997e-06 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504552+0000", 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0445000000000007e-05 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504562+0000", 2024-09-16T03:40:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6819999999999998e-06 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.509553+0000", 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004991021 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.509597+0000", 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4647999999999997e-05 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 52/31 ECSubWrite(tid=44, reqid=client.4615.0:337, at_version=52'36, trim_to=0'0, roll_forward_to=51'34))", 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:31.494440+0000", 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.713527969, 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043633559999999997, 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.494440+0000", 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.494440+0000", 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.494440+0000", 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6e-07 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.495052+0000", 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061194699999999995 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.495065+0000", 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3223e-05 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.495073+0000", 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6550000000000004e-06 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.495155+0000", 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2068000000000001e-05 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.495162+0000", 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4510000000000003e-06 2024-09-16T03:40:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.498762+0000", 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035996579999999999 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.498803+0000", 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1193999999999999e-05 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.309 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:33.310 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:33.308+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:33.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:40:33.411 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:33.410+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:33.452 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:33.453 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:33.453 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:33.453 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:33.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:40:33.505 INFO:teuthology.orchestra.run.smithi076.stderr:marked down osd.13. 2024-09-16T03:40:33.511 INFO:tasks.rados.rados.0.smithi121.stdout:154: done (9 left) 2024-09-16T03:40:33.511 INFO:tasks.rados.rados.0.smithi121.stdout:155: read oid 49 snap -1 2024-09-16T03:40:33.511 INFO:tasks.rados.rados.0.smithi121.stdout:155: expect deleted 2024-09-16T03:40:33.511 INFO:tasks.rados.rados.0.smithi121.stdout:156: delete oid 39 current snap is 12 2024-09-16T03:40:33.511 INFO:tasks.rados.rados.0.smithi121.stdout:143: finishing write tid 4 to smithi12138290-40 2024-09-16T03:40:33.511 INFO:tasks.rados.rados.0.smithi121.stdout:143: oid 40 updating version 0 to 13 2024-09-16T03:40:33.511 INFO:tasks.rados.rados.0.smithi121.stdout:143: oid 40 updating version 13 to 14 2024-09-16T03:40:33.511 INFO:tasks.rados.rados.0.smithi121.stdout:143: oid 40 version 14 is already newer than 12 2024-09-16T03:40:33.511 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 40 v 14 (ObjNum 85 snap 12 seq_num 85) dirty exists 2024-09-16T03:40:33.511 INFO:tasks.rados.rados.0.smithi121.stdout:143: left oid 40 (ObjNum 85 snap 12 seq_num 85) 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:147: finishing write tid 1 to smithi12138290-26 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:147: finishing write tid 2 to smithi12138290-26 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:147: finishing write tid 3 to smithi12138290-26 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:147: finishing write tid 4 to smithi12138290-26 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:148: finishing write tid 1 to smithi12138290-46 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:147: finishing write tid 5 to smithi12138290-26 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:147: oid 26 updating version 0 to 29 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:147: oid 26 updating version 29 to 30 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:147: oid 26 version 30 is already newer than 28 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:147: oid 26 version 30 is already newer than 27 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 26 v 30 (ObjNum 86 snap 12 seq_num 86) dirty exists 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:147: left oid 26 (ObjNum 86 snap 12 seq_num 86) 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:148: finishing write tid 2 to smithi12138290-46 2024-09-16T03:40:33.512 INFO:tasks.rados.rados.0.smithi121.stdout:145: expect (ObjNum 72 snap 6 seq_num 72) 2024-09-16T03:40:33.515 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:33.511+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:33.525 INFO:tasks.thrashosds.thrasher:Removing osd 13, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-16T03:40:33.525 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 13 2024-09-16T03:40:33.570 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:33.571 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:33.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:40:33.615 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:33.614+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:33.622 INFO:tasks.rados.rados.0.smithi121.stdout:148: finishing write tid 3 to smithi12138290-46 2024-09-16T03:40:33.622 INFO:tasks.rados.rados.0.smithi121.stdout:148: finishing write tid 4 to smithi12138290-46 2024-09-16T03:40:33.622 INFO:tasks.rados.rados.0.smithi121.stdout:148: finishing write tid 5 to smithi12138290-46 2024-09-16T03:40:33.622 INFO:tasks.rados.rados.0.smithi121.stdout:148: oid 46 updating version 0 to 12 2024-09-16T03:40:33.622 INFO:tasks.rados.rados.0.smithi121.stdout:148: oid 46 updating version 12 to 14 2024-09-16T03:40:33.622 INFO:tasks.rados.rados.0.smithi121.stdout:148: oid 46 version 14 is already newer than 11 2024-09-16T03:40:33.622 INFO:tasks.rados.rados.0.smithi121.stdout:148: oid 46 version 14 is already newer than 13 2024-09-16T03:40:33.622 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 46 v 14 (ObjNum 87 snap 12 seq_num 87) dirty exists 2024-09-16T03:40:33.623 INFO:tasks.rados.rados.0.smithi121.stdout:148: left oid 46 (ObjNum 87 snap 12 seq_num 87) 2024-09-16T03:40:33.623 INFO:tasks.rados.rados.0.smithi121.stdout:150: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-16T03:40:33.641 INFO:tasks.rados.rados.0.smithi121.stdout:149: expect (ObjNum 73 snap 6 seq_num 73) 2024-09-16T03:40:33.653 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:33.691 INFO:tasks.rados.rados.0.smithi121.stdout:151: finishing write tid 1 to smithi12138290-50 2024-09-16T03:40:33.691 INFO:tasks.rados.rados.0.smithi121.stdout:151: finishing write tid 2 to smithi12138290-50 2024-09-16T03:40:33.691 INFO:tasks.rados.rados.0.smithi121.stdout:152: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-16T03:40:33.710 INFO:tasks.rados.rados.0.smithi121.stdout:151: finishing write tid 3 to smithi12138290-50 2024-09-16T03:40:33.710 INFO:tasks.rados.rados.0.smithi121.stdout:151: finishing write tid 4 to smithi12138290-50 2024-09-16T03:40:33.710 INFO:tasks.rados.rados.0.smithi121.stdout:151: finishing write tid 5 to smithi12138290-50 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:151: oid 50 updating version 0 to 16 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:151: oid 50 updating version 16 to 17 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:151: oid 50 updating version 17 to 19 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:151: oid 50 version 19 is already newer than 18 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 50 v 19 (ObjNum 88 snap 12 seq_num 88) dirty exists 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:151: left oid 50 (ObjNum 88 snap 12 seq_num 88) 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:153: finishing write tid 1 to smithi12138290-18 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:153: finishing write tid 2 to smithi12138290-18 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:143: done (10 left) 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:145: done (9 left) 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:147: done (8 left) 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:148: done (7 left) 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:149: done (6 left) 2024-09-16T03:40:33.711 INFO:tasks.rados.rados.0.smithi121.stdout:150: done (5 left) 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:151: done (4 left) 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:152: done (3 left) 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:156: done (2 left) 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:157: setattr oid 1 current snap is 12 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:153: finishing write tid 3 to smithi12138290-18 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:153: finishing write tid 4 to smithi12138290-18 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:153: finishing write tid 5 to smithi12138290-18 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:153: oid 18 updating version 0 to 45 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:153: oid 18 updating version 45 to 46 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:153: oid 18 updating version 46 to 47 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:153: oid 18 updating version 47 to 48 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 18 v 48 (ObjNum 89 snap 12 seq_num 89) dirty exists 2024-09-16T03:40:33.712 INFO:tasks.rados.rados.0.smithi121.stdout:153: left oid 18 (ObjNum 89 snap 12 seq_num 89) 2024-09-16T03:40:33.713 INFO:tasks.rados.rados.0.smithi121.stdout:153: done (2 left) 2024-09-16T03:40:33.713 INFO:tasks.rados.rados.0.smithi121.stdout:155: done (1 left) 2024-09-16T03:40:33.713 INFO:tasks.rados.rados.0.smithi121.stdout:158: delete oid 32 current snap is 12 2024-09-16T03:40:33.714 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 1 v 46 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:40:33.715 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:33.713+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:33.717 INFO:tasks.rados.rados.0.smithi121.stdout:157: done (1 left) 2024-09-16T03:40:33.717 INFO:tasks.rados.rados.0.smithi121.stdout:158: done (0 left) 2024-09-16T03:40:33.717 INFO:tasks.rados.rados.0.smithi121.stdout:159: read oid 21 snap 3 2024-09-16T03:40:33.717 INFO:tasks.rados.rados.0.smithi121.stdout:159: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T03:40:33.717 INFO:tasks.rados.rados.0.smithi121.stdout:160: delete oid 45 current snap is 12 2024-09-16T03:40:33.719 INFO:tasks.rados.rados.0.smithi121.stdout:160: done (1 left) 2024-09-16T03:40:33.719 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 12 current snap is 12 2024-09-16T03:40:33.719 INFO:tasks.rados.rados.0.smithi121.stdout:161: seq_num 93 ranges {1376256=802816,2179072=638976,2818048=131072} 2024-09-16T03:40:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:33.728 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:33.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:33.728 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:33.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.728 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4341.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.728 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:40:33.728 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 35.463696958, 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59293511399999999, 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4341", 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260707+0000", 2024-09-16T03:40:33.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.66e-06 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260713+0000", 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6150000000000002e-06 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260728+0000", 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5333e-05 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260741+0000", 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260825+0000", 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4147999999999995e-05 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260836+0000", 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0351e-05 2024-09-16T03:40:33.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852485+0000", 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59164876099999997 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852573+0000", 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8352999999999999e-05 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852639+0000", 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853142+0000", 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050253099999999996 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853585+0000", 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000443666 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853615+0000", 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9774e-05 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853640+0000", 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4924999999999999e-05 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.736 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.736 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717039+0000", 2024-09-16T03:40:33.736 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 35.007362508, 2024-09-16T03:40:33.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085512269999999998, 2024-09-16T03:40:33.736 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.736 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717039+0000", 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717039+0000", 2024-09-16T03:40:33.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717040+0000", 2024-09-16T03:40:33.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2099999999999996e-07 2024-09-16T03:40:33.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717044+0000", 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9299999999999996e-06 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717052+0000", 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717058+0000", 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3269999999999997e-06 2024-09-16T03:40:33.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721385+0000", 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043270959999999999 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721439+0000", 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3418999999999998e-05 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721511+0000", 2024-09-16T03:40:33.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2670999999999996e-05 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725365+0000", 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038537179999999999 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725524+0000", 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015862600000000001 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725566+0000", 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2425000000000001e-05 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725591+0000", 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4372e-05 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717067+0000", 2024-09-16T03:40:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 35.007334612000001, 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086214770000000007, 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717067+0000", 2024-09-16T03:40:33.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717067+0000", 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717068+0000", 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.75e-07 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717071+0000", 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5379999999999998e-06 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717077+0000", 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2430000000000002e-06 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717082+0000", 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5879999999999999e-06 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721746+0000", 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046643919999999998 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721841+0000", 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4192999999999998e-05 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.721969+0000", 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012844500000000001 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725599+0000", 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036301150000000002 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725638+0000", 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9110000000000003e-05 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725665+0000", 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-09-16T03:40:33.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725689+0000", 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3546000000000001e-05 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 35.007288140999997, 2024-09-16T03:40:33.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093665129999999999, 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.06e-07 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717170+0000", 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5937000000000003e-05 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717182+0000", 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1844999999999999e-05 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717188+0000", 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.054e-06 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.724763+0000", 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075747119999999999 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.724809+0000", 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5933999999999998e-05 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.724882+0000", 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3650000000000001e-05 2024-09-16T03:40:33.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726043+0000", 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011606260000000001 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726385+0000", 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034193800000000002 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726443+0000", 2024-09-16T03:40:33.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8251999999999997e-05 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726480+0000", 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6959000000000003e-05 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 35.007198899999999, 2024-09-16T03:40:33.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095750269999999998, 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7800000000000001e-07 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717208+0000", 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.109e-06 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717215+0000", 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0210000000000002e-06 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717226+0000", 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0934000000000001e-05 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725719+0000", 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084932659999999993 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725774+0000", 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5260000000000003e-05 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725850+0000", 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5907000000000004e-05 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726495+0000", 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064497100000000004 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726716+0000", 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022091799999999999 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726744+0000", 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8561e-05 2024-09-16T03:40:33.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726778+0000", 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3501999999999999e-05 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717235+0000", 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 35.007166701999999, 2024-09-16T03:40:33.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097518989999999996, 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717235+0000", 2024-09-16T03:40:33.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717235+0000", 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717236+0000", 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8400000000000005e-07 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717240+0000", 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717247+0000", 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4939999999999997e-06 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717252+0000", 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726072+0000", 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088197359999999999 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726119+0000", 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6662000000000001e-05 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726196+0000", 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6958000000000004e-05 2024-09-16T03:40:33.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726593+0000", 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000397272 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726935+0000", 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034162800000000002 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726964+0000", 2024-09-16T03:40:33.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9268000000000002e-05 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726987+0000", 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3096e-05 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:33.766 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.573132958000002, 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094025229999999994, 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 162 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:33.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151271+0000", 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2510000000000001e-06 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151275+0000", 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4199999999999999e-06 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151278+0000", 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8669999999999998e-06 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.156802+0000", 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055245119999999997 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.156852+0000", 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0195999999999998e-05 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.156927+0000", 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4192999999999999e-05 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159094+0000", 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021670740000000002 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160511+0000", 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001417583 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160599+0000", 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7965000000000002e-05 2024-09-16T03:40:33.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160671+0000", 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2113999999999994e-05 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.573120090000003, 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094948329999999994, 2024-09-16T03:40:33.772 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 163 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.11e-07 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151284+0000", 2024-09-16T03:40:33.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2680000000000001e-06 2024-09-16T03:40:33.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151287+0000", 2024-09-16T03:40:33.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2820000000000001e-06 2024-09-16T03:40:33.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151291+0000", 2024-09-16T03:40:33.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2100000000000002e-06 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157351+0000", 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060598270000000003 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157437+0000", 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6279999999999994e-05 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157555+0000", 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118377 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159098+0000", 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015432499999999999 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160709+0000", 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016109799999999999 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160743+0000", 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3432000000000001e-05 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160777+0000", 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3716e-05 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151306+0000", 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.573095369999997, 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01219754, 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151306+0000", 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151306+0000", 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151307+0000", 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4999999999999999e-07 2024-09-16T03:40:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151364+0000", 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7055000000000002e-05 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151371+0000", 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0550000000000002e-06 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151374+0000", 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.534e-06 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157903+0000", 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065283620000000002 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157969+0000", 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6211999999999996e-05 2024-09-16T03:40:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158097+0000", 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127737 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159104+0000", 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010076919999999999 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163354+0000", 2024-09-16T03:40:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042495579999999996 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163439+0000", 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5276000000000001e-05 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163504+0000", 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4808999999999996e-05 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.573021031000003, 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012275085, 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 165 2024-09-16T03:40:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3799999999999998e-07 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151384+0000", 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5639999999999999e-06 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151388+0000", 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9280000000000003e-06 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151391+0000", 2024-09-16T03:40:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0699999999999998e-06 2024-09-16T03:40:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158639+0000", 2024-09-16T03:40:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072479620000000002 2024-09-16T03:40:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158736+0000", 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7435000000000005e-05 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158872+0000", 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000136119 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159390+0000", 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051745999999999997 2024-09-16T03:40:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163597+0000", 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042074870000000002 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163631+0000", 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4057000000000003e-05 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163656+0000", 2024-09-16T03:40:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4664999999999998e-05 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:180 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.282924+0000", 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.441478033999999, 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092163079999999994, 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 180 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282924+0000", 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282924+0000", 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282924+0000", 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9099999999999999e-07 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282929+0000", 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3449999999999999e-06 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282933+0000", 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6720000000000003e-06 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.282936+0000", 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7580000000000001e-06 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.284410+0000", 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001474265 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.284472+0000", 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1357000000000002e-05 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.284588+0000", 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011673 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286974+0000", 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023862110000000001 2024-09-16T03:40:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292099+0000", 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051242029999999999 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292123+0000", 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.402e-05 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292140+0000", 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7356000000000001e-05 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:181 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283040+0000", 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.44136177, 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092435739999999992, 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 181 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283040+0000", 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283040+0000", 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283041+0000", 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5499999999999998e-07 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283044+0000", 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8850000000000001e-06 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283052+0000", 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6240000000000002e-06 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283057+0000", 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.481e-06 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.284980+0000", 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019231160000000001 2024-09-16T03:40:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.285058+0000", 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8047999999999995e-05 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.285136+0000", 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8516000000000005e-05 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286980+0000", 2024-09-16T03:40:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018439769999999999 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292190+0000", 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052099920000000001 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292243+0000", 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2831000000000003e-05 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292284+0000", 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0549000000000001e-05 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:182 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283109+0000", 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.441293223999999, 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092439459999999994, 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 182 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283109+0000", 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283109+0000", 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283109+0000", 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7300000000000002e-07 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283183+0000", 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3855000000000004e-05 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283188+0000", 2024-09-16T03:40:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1510000000000003e-06 2024-09-16T03:40:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283191+0000", 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.285359+0000", 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021678969999999998 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.285433+0000", 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3707000000000004e-05 2024-09-16T03:40:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.285550+0000", 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116686 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288022+0000", 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024726729999999999 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292310+0000", 2024-09-16T03:40:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042879900000000002 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292335+0000", 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4355999999999999e-05 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292353+0000", 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7983999999999999e-05 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:183 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283197+0000", 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.441204442, 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093764810000000007, 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 183 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283197+0000", 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283197+0000", 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283198+0000", 2024-09-16T03:40:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1199999999999999e-07 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283200+0000", 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1780000000000002e-06 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283203+0000", 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.41e-06 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283206+0000", 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5730000000000002e-06 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286007+0000", 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028007969999999998 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286078+0000", 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1606000000000005e-05 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286188+0000", 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000109978 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288028+0000", 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018400619999999999 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292533+0000", 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045046419999999997 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292556+0000", 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2733000000000001e-05 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292574+0000", 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.819e-05 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:184 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.441192616999999, 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094925180000000001, 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 184 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0699999999999999e-07 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283211+0000", 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8810000000000001e-06 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283214+0000", 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1700000000000001e-06 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283219+0000", 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1989999999999999e-06 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286487+0000", 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032681239999999999 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286562+0000", 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5328999999999997e-05 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286677+0000", 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114856 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289031+0000", 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023536310000000001 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292619+0000", 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035884580000000001 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292667+0000", 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8282000000000003e-05 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292702+0000", 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4381000000000002e-05 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.441173792999997, 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095539089999999993, 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 185 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4600000000000001e-07 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283260+0000", 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1488000000000001e-05 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283264+0000", 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283267+0000", 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0299999999999998e-06 2024-09-16T03:40:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287022+0000", 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037554400000000001 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287090+0000", 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7329999999999994e-05 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287210+0000", 2024-09-16T03:40:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119838 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289036+0000", 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001826814 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292735+0000", 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036988400000000001 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292762+0000", 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7036e-05 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292782+0000", 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9690999999999998e-05 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283295+0000", 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.441106640999998, 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0096111779999999997, 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 186 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283295+0000", 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283295+0000", 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283296+0000", 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0299999999999999e-07 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283299+0000", 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6500000000000002e-06 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283304+0000", 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8790000000000002e-06 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283307+0000", 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.061e-06 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287556+0000", 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042482650000000002 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287625+0000", 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8989e-05 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287736+0000", 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111983 2024-09-16T03:40:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289043+0000", 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013063020000000001 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292823+0000", 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037805619999999999 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292871+0000", 2024-09-16T03:40:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7482999999999999e-05 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292906+0000", 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5500999999999998e-05 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.441089187000003, 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0096710279999999999, 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283315+0000", 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.227e-06 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283322+0000", 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2119999999999997e-06 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283325+0000", 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9000000000000002e-06 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288058+0000", 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047327790000000003 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288104+0000", 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5788999999999998e-05 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288174+0000", 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9769000000000002e-05 2024-09-16T03:40:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289048+0000", 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087418399999999996 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292940+0000", 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038922129999999998 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292965+0000", 2024-09-16T03:40:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4929000000000001e-05 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292984+0000", 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8695999999999999e-05 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283335+0000", 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.441066468000002, 2024-09-16T03:40:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097754880000000006, 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 188 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283335+0000", 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283335+0000", 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283336+0000", 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8599999999999999e-07 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283372+0000", 2024-09-16T03:40:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6279000000000003e-05 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283376+0000", 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.454e-06 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283379+0000", 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288401+0000", 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050215930000000004 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288476+0000", 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4667999999999997e-05 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288596+0000", 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120439 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289367+0000", 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00077095699999999996 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293025+0000", 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036582390000000002 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293075+0000", 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0050999999999999e-05 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293111+0000", 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5525999999999998e-05 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283384+0000", 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.441017539000001, 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097953789999999999, 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 189 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283384+0000", 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283384+0000", 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283385+0000", 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7799999999999997e-07 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283387+0000", 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2330000000000001e-06 2024-09-16T03:40:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283391+0000", 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.878e-06 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283393+0000", 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.525e-06 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289066+0000", 2024-09-16T03:40:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056729689999999999 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289107+0000", 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0513999999999999e-05 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289176+0000", 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9481999999999996e-05 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289693+0000", 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051673699999999997 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293139+0000", 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034465229999999999 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293163+0000", 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3811999999999998e-05 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293180+0000", 2024-09-16T03:40:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6427999999999999e-05 2024-09-16T03:40:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:33.853 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:33.854 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:33.855 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:33.855 INFO:tasks.rados.rados.0.smithi121.stdout:161: writing smithi12138290-12 from 1376256 to 2179072 tid 1 2024-09-16T03:40:33.855 INFO:tasks.rados.rados.0.smithi121.stdout:161: writing smithi12138290-12 from 2179072 to 2818048 tid 2 2024-09-16T03:40:33.856 INFO:tasks.rados.rados.0.smithi121.stdout:161: writing smithi12138290-12 from 2818048 to 2949120 tid 3 2024-09-16T03:40:33.856 INFO:tasks.rados.rados.0.smithi121.stdout:162: read oid 35 snap -1 2024-09-16T03:40:33.856 INFO:tasks.rados.rados.0.smithi121.stdout:162: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:40:33.856 INFO:tasks.rados.rados.0.smithi121.stdout:163: snap_remove snap 1 2024-09-16T03:40:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:33.857 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:33.857 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:33.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:40:33.858 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:33.858 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:33.859 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:33.853+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:33.862 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T03:40:33.860+0000 7f011120c640 -1 mon.a@0(leader).osd e54 definitely_dead 0 2024-09-16T03:40:33.863 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:33.879 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:33.880 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:33.896 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:33.896 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:33.912 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:33.912 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:33.929 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:33.929 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:33.945 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:33.946 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:33.955 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:33.953+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:33.962 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:33.962 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:33.978 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:33.978 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:33.978 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-09-16T03:40:33.995 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:33.995 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:34.001 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:34.001 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:34.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:40:34.013 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:34.013 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:40:34.016 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:40:34.055 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:34.054+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:34.115 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:34.115 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:34.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:40:34.156 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:34.154+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:34.225 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:34.226 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:34.234 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:40:34.256 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:34.254+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:34.338 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:34.339 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:34.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:40:34.358 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:34.356+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:34.450 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:34.450 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:34.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:40:34.459 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:34.456+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:34.506 INFO:teuthology.orchestra.run.smithi076.stderr:marked out osd.13. 2024-09-16T03:40:34.512 INFO:tasks.rados.rados.0.smithi121.stdout:163: done (3 left) 2024-09-16T03:40:34.513 INFO:tasks.rados.rados.0.smithi121.stdout:164: read oid 13 snap -1 2024-09-16T03:40:34.513 INFO:tasks.rados.rados.0.smithi121.stdout:164: expect (ObjNum 82 snap 12 seq_num 82) 2024-09-16T03:40:34.513 INFO:tasks.rados.rados.0.smithi121.stdout:165: rollback oid 37 current snap is 12 2024-09-16T03:40:34.513 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 37 to 6 2024-09-16T03:40:34.513 INFO:tasks.rados.rados.0.smithi121.stdout:162: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:40:34.522 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:40:34.527 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.13 2024-09-16T03:40:34.527 INFO:tasks.thrashosds.thrasher:remote for osd 13 is ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:40:34.528 DEBUG:teuthology.orchestra.run.smithi103:> type ceph-objectstore-tool 2024-09-16T03:40:34.535 INFO:tasks.rados.rados.0.smithi121.stdout:162: done (4 left) 2024-09-16T03:40:34.535 INFO:tasks.rados.rados.0.smithi121.stdout:166: read oid 28 snap -1 2024-09-16T03:40:34.535 INFO:tasks.rados.rados.0.smithi121.stdout:166: expect (ObjNum 73 snap 6 seq_num 73) 2024-09-16T03:40:34.535 INFO:tasks.rados.rados.0.smithi121.stdout:167: snap_create 2024-09-16T03:40:34.535 INFO:tasks.rados.rados.0.smithi121.stdout:165: finishing rollback tid 0 to smithi12138290-37 2024-09-16T03:40:34.535 INFO:tasks.rados.rados.0.smithi121.stdout:165: finishing rollback tid 1 to smithi12138290-37 2024-09-16T03:40:34.535 INFO:tasks.rados.rados.0.smithi121.stdout:165: finishing rollback tid 2 to smithi12138290-37 2024-09-16T03:40:34.535 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 37 v 34 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-16T03:40:34.535 INFO:tasks.rados.rados.0.smithi121.stdout:164: expect (ObjNum 82 snap 12 seq_num 82) 2024-09-16T03:40:34.545 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T03:40:34.546 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/journal --op list-pgs 2024-09-16T03:40:34.559 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:34.557+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:34.578 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:34.578 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:34.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:40:34.660 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:34.658+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:34.733 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:34.733 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:34.742 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:40:34.760 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:34.759+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:34.846 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:34.846 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:34.855 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:40:34.898 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:34.897+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:34.958 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:34.959 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:34.962 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:34.960+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:34.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:40:35.063 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:35.061+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:35.112 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:35.112 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:35.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:40:35.165 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:35.164+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:35.224 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:35.224 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:35.232 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:40:35.274 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:35.272+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:35.366 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:35.364+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:35.376 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:35.376 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:35.385 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:40:35.466 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:35.464+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:35.488 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:35.488 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:35.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:40:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T03:40:35.514 INFO:tasks.rados.rados.0.smithi121.stdout:164: done (5 left) 2024-09-16T03:40:35.514 INFO:tasks.rados.rados.0.smithi121.stdout:165: done (4 left) 2024-09-16T03:40:35.514 INFO:tasks.rados.rados.0.smithi121.stdout:167: done (3 left) 2024-09-16T03:40:35.514 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 5 current snap is 13 2024-09-16T03:40:35.514 INFO:tasks.rados.rados.0.smithi121.stdout:168: seq_num 94 ranges {0=425984,425984=786432,1212416=376832} 2024-09-16T03:40:35.517 INFO:tasks.rados.rados.0.smithi121.stdout:168: writing smithi12138290-5 from 0 to 425984 tid 1 2024-09-16T03:40:35.522 INFO:tasks.rados.rados.0.smithi121.stdout:168: writing smithi12138290-5 from 425984 to 1212416 tid 2 2024-09-16T03:40:35.524 INFO:tasks.rados.rados.0.smithi121.stdout:168: writing smithi12138290-5 from 1212416 to 1589248 tid 3 2024-09-16T03:40:35.524 INFO:tasks.rados.rados.0.smithi121.stdout:169: delete oid 38 current snap is 13 2024-09-16T03:40:35.530 INFO:tasks.rados.rados.0.smithi121.stdout:159: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T03:40:35.541 INFO:tasks.rados.rados.0.smithi121.stdout:166: expect (ObjNum 73 snap 6 seq_num 73) 2024-09-16T03:40:35.572 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:35.571+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:35.600 INFO:tasks.rados.rados.0.smithi121.stdout:168: finishing write tid 1 to smithi12138290-5 2024-09-16T03:40:35.600 INFO:tasks.rados.rados.0.smithi121.stdout:168: finishing write tid 2 to smithi12138290-5 2024-09-16T03:40:35.600 INFO:tasks.rados.rados.0.smithi121.stdout:159: done (4 left) 2024-09-16T03:40:35.600 INFO:tasks.rados.rados.0.smithi121.stdout:166: done (3 left) 2024-09-16T03:40:35.600 INFO:tasks.rados.rados.0.smithi121.stdout:169: done (2 left) 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:170: copy_from oid 26 from oid 50 current snap is 13 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:168: finishing write tid 3 to smithi12138290-5 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:168: finishing write tid 4 to smithi12138290-5 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:168: finishing write tid 5 to smithi12138290-5 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:168: oid 5 updating version 0 to 7 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:168: oid 5 updating version 7 to 9 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:168: oid 5 version 9 is already newer than 8 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:168: oid 5 version 9 is already newer than 6 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 5 v 9 (ObjNum 94 snap 13 seq_num 94) dirty exists 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:168: left oid 5 (ObjNum 94 snap 13 seq_num 94) 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:161: finishing write tid 1 to smithi12138290-12 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:161: finishing write tid 2 to smithi12138290-12 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:161: finishing write tid 3 to smithi12138290-12 2024-09-16T03:40:35.601 INFO:tasks.rados.rados.0.smithi121.stdout:161: finishing write tid 4 to smithi12138290-12 2024-09-16T03:40:35.602 INFO:tasks.rados.rados.0.smithi121.stdout:161: finishing write tid 5 to smithi12138290-12 2024-09-16T03:40:35.602 INFO:tasks.rados.rados.0.smithi121.stdout:161: oid 12 updating version 0 to 31 2024-09-16T03:40:35.602 INFO:tasks.rados.rados.0.smithi121.stdout:161: oid 12 version 31 is already newer than 28 2024-09-16T03:40:35.602 INFO:tasks.rados.rados.0.smithi121.stdout:161: oid 12 version 31 is already newer than 30 2024-09-16T03:40:35.602 INFO:tasks.rados.rados.0.smithi121.stdout:161: oid 12 version 31 is already newer than 29 2024-09-16T03:40:35.602 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 31 (ObjNum 93 snap 12 seq_num 93) dirty exists 2024-09-16T03:40:35.602 INFO:tasks.rados.rados.0.smithi121.stdout:161: left oid 12 (ObjNum 93 snap 12 seq_num 93) 2024-09-16T03:40:35.602 INFO:tasks.rados.rados.0.smithi121.stdout:161: done (2 left) 2024-09-16T03:40:35.602 INFO:tasks.rados.rados.0.smithi121.stdout:168: done (1 left) 2024-09-16T03:40:35.602 INFO:tasks.rados.rados.0.smithi121.stdout:171: snap_create 2024-09-16T03:40:35.613 INFO:tasks.rados.rados.0.smithi121.stdout:170: finishing copy_from racing read to smithi12138290-26 2024-09-16T03:40:35.617 INFO:tasks.rados.rados.0.smithi121.stdout:170: finishing copy_from to smithi12138290-26 2024-09-16T03:40:35.617 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 26 v 19 (ObjNum 88 snap 12 seq_num 88) dirty exists 2024-09-16T03:40:35.625 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:35.625 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:35.634 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:40:35.675 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:35.674+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:35.769 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:35.768+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:35.777 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:35.778 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:35.787 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:40:35.870 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:35.868+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:35.891 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:35.891 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:35.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:40:35.971 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:35.970+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:36.001 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:36.001 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:36.010 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:40:36.017 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:36.072 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:36.070+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:36.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:40:36.139 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:36.139 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:36.156 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:36.157 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:36.173 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:36.174 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:36.171+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:36.190 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:36.191 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:36.206 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:36.224 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:36.225 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:36.241 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:36.242 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:36.258 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:36.272 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:36.273 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:36.273 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:36.273 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:36.274 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:36.272+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:36.274 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:36.275 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:36.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:40:36.291 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:36.291 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:36.308 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:36.308 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:36.324 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:36.325 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:36.340 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-09-16T03:40:36.375 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:36.374+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:36.424 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:36.425 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:36.425 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:36.425 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:36.425 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:36.433 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:40:36.471 INFO:tasks.rados.rados.0.smithi121.stdout:170: done (1 left) 2024-09-16T03:40:36.471 INFO:tasks.rados.rados.0.smithi121.stdout:171: done (0 left) 2024-09-16T03:40:36.471 INFO:tasks.rados.rados.0.smithi121.stdout:172: read oid 11 snap -1 2024-09-16T03:40:36.471 INFO:tasks.rados.rados.0.smithi121.stdout:172: expect (ObjNum 77 snap 11 seq_num 77) 2024-09-16T03:40:36.471 INFO:tasks.rados.rados.0.smithi121.stdout:173: snap_create 2024-09-16T03:40:36.476 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:36.474+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:36.482 INFO:tasks.rados.rados.0.smithi121.stdout:172: expect (ObjNum 77 snap 11 seq_num 77) 2024-09-16T03:40:36.533 INFO:teuthology.orchestra.run.smithi103.stdout:3.fs1 2024-09-16T03:40:36.533 INFO:teuthology.orchestra.run.smithi103.stdout:3.es0 2024-09-16T03:40:36.533 INFO:teuthology.orchestra.run.smithi103.stdout:3.as4 2024-09-16T03:40:36.533 INFO:teuthology.orchestra.run.smithi103.stdout:3.9s0 2024-09-16T03:40:36.533 INFO:teuthology.orchestra.run.smithi103.stdout:3.7s1 2024-09-16T03:40:36.533 INFO:teuthology.orchestra.run.smithi103.stdout:3.4s1 2024-09-16T03:40:36.533 INFO:teuthology.orchestra.run.smithi103.stdout:2.6 2024-09-16T03:40:36.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:36.542+0000 7fcaa5a40640 0 -- 172.21.15.76:0/3818404340 >> [v2:172.21.15.119:6800/1147733023,v1:172.21.15.119:6801/1147733023] conn(0x7fcaa0219080 msgr2=0x7fcaa0219500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:36.560 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:36.561 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:36.577 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:36.577 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:36.577 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:36.574+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:36.580 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:36.580 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:36.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:36.580 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:36.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:39 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'25, mlcod=25'22)", 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 38.259899900000001, 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042816670000000003, 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317957+0000", 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.54e-07 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318024+0000", 2024-09-16T03:40:36.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6730000000000007e-05 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318033+0000", 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1589999999999999e-06 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318038+0000", 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8040000000000001e-06 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318080+0000", 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2475000000000002e-05 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318093+0000", 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3315e-05 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.322203+0000", 2024-09-16T03:40:36.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041094620000000004 2024-09-16T03:40:36.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.322238+0000", 2024-09-16T03:40:36.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4968000000000001e-05 2024-09-16T03:40:36.585 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.585 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.585 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:164 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'118, mlcod=25'113)", 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.419622765, 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047753090000000002, 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:36.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158234+0000", 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8100000000000003e-07 2024-09-16T03:40:36.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158294+0000", 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0359999999999998e-05 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158301+0000", 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3159999999999999e-06 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158305+0000", 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3900000000000002e-06 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158343+0000", 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7793000000000002e-05 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158361+0000", 2024-09-16T03:40:36.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8230999999999999e-05 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.162979+0000", 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046178230000000001 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.163009+0000", 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9915000000000001e-05 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.590 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:178 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'128, mlcod=25'125)", 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.294217901000003, 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058842529999999999, 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283639+0000", 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283643+0000", 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3980000000000001e-06 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283652+0000", 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6430000000000001e-06 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283656+0000", 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7339999999999999e-06 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283698+0000", 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1785000000000001e-05 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283715+0000", 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6555e-05 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289498+0000", 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0057832869999999998 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289522+0000", 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4290999999999998e-05 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:179 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'129, mlcod=25'125)", 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:36.595 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.293746771999999, 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066372109999999996, 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284110+0000", 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.89e-07 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284175+0000", 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5667000000000007e-05 2024-09-16T03:40:36.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284186+0000", 2024-09-16T03:40:36.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.009e-05 2024-09-16T03:40:36.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284190+0000", 2024-09-16T03:40:36.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7600000000000002e-06 2024-09-16T03:40:36.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284222+0000", 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1888999999999999e-05 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284234+0000", 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.225e-05 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290722+0000", 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064871909999999998 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290746+0000", 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4875000000000002e-05 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:180 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'130, mlcod=25'125)", 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.293132462999999, 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072760109999999998, 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.68e-07 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284729+0000", 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4009999999999999e-06 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284738+0000", 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6940000000000007e-06 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284743+0000", 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5549999999999996e-06 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286547+0000", 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0018044770000000001 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286567+0000", 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9272000000000001e-05 2024-09-16T03:40:36.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.291958+0000", 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053911540000000004 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292000+0000", 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.189e-05 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:181 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'131, mlcod=25'126)", 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.292591913000003, 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067648710000000004, 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:36.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285265+0000", 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6100000000000001e-07 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285269+0000", 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5020000000000003e-06 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285279+0000", 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6879999999999994e-06 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285286+0000", 2024-09-16T03:40:36.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4590000000000001e-06 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286842+0000", 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015560000000000001 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286854+0000", 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1959e-05 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292001+0000", 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051468859999999998 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292029+0000", 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7815999999999999e-05 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:182 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'132, mlcod=25'126)", 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.292147192999998, 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0063484680000000003, 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2200000000000004e-07 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285737+0000", 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7886999999999999e-05 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285747+0000", 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9459999999999993e-06 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285752+0000", 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:40:36.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287024+0000", 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012715409999999999 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287042+0000", 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8919999999999998e-05 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292031+0000", 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049888179999999999 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292057+0000", 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6057000000000001e-05 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:183 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'133, mlcod=25'126)", 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.291452098999997, 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060351210000000001, 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:36.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286405+0000", 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4300000000000001e-06 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286429+0000", 2024-09-16T03:40:36.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3459999999999999e-05 2024-09-16T03:40:36.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286439+0000", 2024-09-16T03:40:36.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0105999999999999e-05 2024-09-16T03:40:36.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286443+0000", 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.527e-06 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289328+0000", 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028841940000000001 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289340+0000", 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1861e-05 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292411+0000", 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030717399999999999 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292439+0000", 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7803000000000001e-05 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.620 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:184 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'134, mlcod=25'126)", 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.291037731999999, 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056342509999999998, 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286819+0000", 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5499999999999998e-07 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286823+0000", 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2889999999999999e-06 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286833+0000", 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4769999999999992e-06 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286839+0000", 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6599999999999998e-06 2024-09-16T03:40:36.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289572+0000", 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002733161 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289593+0000", 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0494999999999999e-05 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292440+0000", 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028470660000000001 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292453+0000", 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2548000000000001e-05 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:185 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'135, mlcod=25'126)", 2024-09-16T03:40:36.625 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.290473980000002, 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050824039999999996, 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287383+0000", 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287410+0000", 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6985e-05 2024-09-16T03:40:36.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287419+0000", 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6879999999999994e-06 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287424+0000", 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6829999999999999e-06 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289884+0000", 2024-09-16T03:40:36.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002459639 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289917+0000", 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3303999999999998e-05 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292453+0000", 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025360769999999999 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292464+0000", 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1364e-05 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:186 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'136, mlcod=25'126)", 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.289959347999996, 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045805179999999996, 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.630 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:36.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3200000000000005e-07 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287902+0000", 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.634e-06 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287912+0000", 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7739999999999999e-06 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287916+0000", 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6099999999999999e-06 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290274+0000", 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023574519999999999 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290289+0000", 2024-09-16T03:40:36.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5481e-05 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292466+0000", 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021765439999999999 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292477+0000", 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:204 1.0 e26/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'150, mlcod=25'145)", 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.166591177999997, 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051027959999999997, 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:36.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411272+0000", 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1029999999999998e-06 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411280+0000", 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3129999999999999e-06 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411287+0000", 2024-09-16T03:40:36.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2570000000000001e-06 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411347+0000", 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0214000000000003e-05 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411361+0000", 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3735e-05 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416329+0000", 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049679889999999999 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416368+0000", 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8634999999999999e-05 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:205 1.0 e26/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'151, mlcod=26'147)", 2024-09-16T03:40:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.16605191, 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004582226, 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411805+0000", 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2699999999999999e-07 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411808+0000", 2024-09-16T03:40:36.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6710000000000001e-06 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411815+0000", 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9480000000000002e-06 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411820+0000", 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0270000000000003e-06 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411851+0000", 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0474e-05 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411862+0000", 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.171e-05 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416370+0000", 2024-09-16T03:40:36.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045072810000000001 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416386+0000", 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6588000000000001e-05 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es2 37/31 ECSubWrite(tid=26, reqid=client.4615.0:218, at_version=37'8, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.556575+0000", 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 20.021280753999999, 2024-09-16T03:40:36.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044039439999999999, 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556575+0000", 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556575+0000", 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556578+0000", 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8490000000000002e-06 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556597+0000", 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8513999999999999e-05 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556607+0000", 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0309000000000001e-05 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556616+0000", 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3889999999999994e-06 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556677+0000", 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1066000000000001e-05 2024-09-16T03:40:36.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556686+0000", 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.952e-06 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560938+0000", 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042516860000000002 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560979+0000", 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1179e-05 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.5s6 42/31 ECSubWrite(tid=23, reqid=osd.8.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.555040+0000", 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.022816328999999, 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005476687, 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:36.649 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555040+0000", 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555040+0000", 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555041+0000", 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0500000000000002e-07 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555051+0000", 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0305e-05 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555063+0000", 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2002e-05 2024-09-16T03:40:36.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555069+0000", 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5889999999999999e-06 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555132+0000", 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2514000000000005e-05 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:36.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555142+0000", 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0118999999999999e-05 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560482+0000", 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053403239999999996 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560516+0000", 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3929e-05 2024-09-16T03:40:36.653 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 42/31 ECSubWrite(tid=73, reqid=osd.0.0:73, at_version=42'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.021936981, 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045524069999999996, 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:36.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7700000000000001e-07 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555946+0000", 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6560999999999999e-05 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555959+0000", 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3042e-05 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555968+0000", 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1079999999999999e-06 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556053+0000", 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4975999999999994e-05 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556063+0000", 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6120000000000005e-06 2024-09-16T03:40:36.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560429+0000", 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043663369999999996 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560471+0000", 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2493999999999997e-05 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 42/31 ECSubWrite(tid=74, reqid=osd.0.0:73, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.014981596, 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064234579999999999, 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:40:36.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562875+0000", 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562885+0000", 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0069e-05 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562897+0000", 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1619e-05 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562903+0000", 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.652e-06 2024-09-16T03:40:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562965+0000", 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1576999999999997e-05 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562975+0000", 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0314e-05 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:36.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569190+0000", 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062148569999999998 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569298+0000", 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010764600000000001 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/31 ECSubWrite(tid=89, reqid=osd.0.0:88, at_version=0'0, trim_to=0'0, roll_forward_to=53'24))", 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.500490+0000", 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.0773663710000001, 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048251589999999999, 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500490+0000", 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500490+0000", 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500491+0000", 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9999999999999995e-07 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500506+0000", 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5305e-05 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500519+0000", 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.349e-05 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500531+0000", 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1719000000000001e-05 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500605+0000", 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3694000000000003e-05 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500615+0000", 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9080000000000007e-06 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505262+0000", 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046472570000000001 2024-09-16T03:40:36.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505315+0000", 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2785999999999999e-05 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.4s6 53/31 ECSubWrite(tid=56, reqid=client.4615.0:370, at_version=53'16, trim_to=0'0, roll_forward_to=53'13))", 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.533706+0000", 2024-09-16T03:40:36.668 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.0441500709999998, 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046483239999999997, 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533706+0000", 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533706+0000", 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533707+0000", 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7499999999999999e-07 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533723+0000", 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6269e-05 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533748+0000", 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4736000000000002e-05 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533757+0000", 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2739999999999996e-06 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533826+0000", 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9439999999999999e-05 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533839+0000", 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2962999999999999e-05 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.538317+0000", 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044773130000000001 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.538354+0000", 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7554e-05 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.4s6 53/31 ECSubWrite(tid=57, reqid=client.4615.0:370, at_version=0'0, trim_to=0'0, roll_forward_to=53'16))", 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.539071+0000", 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.0387846100000004, 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045011890000000001, 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539071+0000", 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539071+0000", 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539072+0000", 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3099999999999996e-07 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539082+0000", 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7869999999999996e-06 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539097+0000", 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4498000000000001e-05 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539104+0000", 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9619999999999993e-06 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539165+0000", 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0754999999999997e-05 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539175+0000", 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5449999999999992e-06 2024-09-16T03:40:36.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.543533+0000", 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043584690000000002 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.543573+0000", 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9341999999999997e-05 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.677 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:36.678 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:36.678 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:40:36.680 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:40:36.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:40:36.681 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:36.677+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:36.780 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:36.779+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:36.785 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:36.785 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:36.785 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:36.785 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:36.793 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:40:36.881 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:36.879+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:36.897 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:36.897 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:36.897 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:36.897 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:36.897 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:36.906 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:40:36.982 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:36.981+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:37.011 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:231 3.7s0 3:e441e0cd:::smithi12138290-34:head [append 0~737280] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.548164+0000", 2024-09-16T03:40:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.460301839, 2024-09-16T03:40:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074938289999999996, 2024-09-16T03:40:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 231 2024-09-16T03:40:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.548164+0000", 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.548164+0000", 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.548166+0000", 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4929999999999999e-06 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549023+0000", 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00085696500000000001 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549041+0000", 2024-09-16T03:40:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8612000000000001e-05 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549051+0000", 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5389999999999996e-06 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549108+0000", 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6780999999999997e-05 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549115+0000", 2024-09-16T03:40:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3690000000000001e-06 2024-09-16T03:40:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550938+0000", 2024-09-16T03:40:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001823376 2024-09-16T03:40:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.551048+0000", 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010907499999999999 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.552071+0000", 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0010230149999999999 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555488+0000", 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034173379999999998 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555573+0000", 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4932000000000006e-05 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555658+0000", 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5334000000000007e-05 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:232 3.7s0 3:e441e0cd:::smithi12138290-34:head [append 0~376832] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.550228+0000", 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.458238153, 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058243390000000004, 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 232 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550228+0000", 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550228+0000", 2024-09-16T03:40:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550229+0000", 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2999999999999999e-07 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550657+0000", 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042805700000000003 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550670+0000", 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2796000000000001e-05 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550678+0000", 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9219999999999997e-06 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.553144+0000", 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024667650000000001 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.553239+0000", 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4949000000000002e-05 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.553968+0000", 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00072873999999999997 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555765+0000", 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001796874 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556000+0000", 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000235443 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556052+0000", 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1863000000000001e-05 2024-09-16T03:40:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:233 3.7s0 3:e441e0cd:::smithi12138290-34:head [setxattr _header (56) in=7b] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.550693+0000", 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.457773546999999, 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062768190000000003, 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 233 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550693+0000", 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550693+0000", 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550693+0000", 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6700000000000003e-07 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550697+0000", 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4170000000000001e-06 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550703+0000", 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6830000000000002e-06 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550709+0000", 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1660000000000002e-06 2024-09-16T03:40:37.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.554595+0000", 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038863970000000002 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.554682+0000", 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7299999999999994e-05 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555082+0000", 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00039952299999999999 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555766+0000", 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00068426399999999997 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556891+0000", 2024-09-16T03:40:37.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001124508 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556969+0000", 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8893999999999994e-05 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:234 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:37.031 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.550720+0000", 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.45774613, 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075177949999999999, 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 234 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.032 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550720+0000", 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550720+0000", 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550721+0000", 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9200000000000001e-07 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550723+0000", 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2409999999999998e-06 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550730+0000", 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8750000000000002e-06 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550736+0000", 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8579999999999996e-06 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556094+0000", 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053587419999999997 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556171+0000", 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7232999999999997e-05 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.557335+0000", 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011640000000000001 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.557405+0000", 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9979000000000002e-05 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.558238+0000", 2024-09-16T03:40:37.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000832375 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.0s1 42/31 ECSubWrite(tid=20, reqid=client.4615.0:251, at_version=42'11, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.564475+0000", 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.443990914, 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050564989999999999, 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.037 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564475+0000", 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564475+0000", 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564476+0000", 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7300000000000002e-07 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564523+0000", 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7376999999999997e-05 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564537+0000", 2024-09-16T03:40:37.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4503e-05 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564556+0000", 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8417e-05 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564627+0000", 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1507000000000005e-05 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564640+0000", 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2629000000000001e-05 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569498+0000", 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048575579999999997 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569532+0000", 2024-09-16T03:40:37.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4235e-05 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpRead(3.ds2 42/31 ECSubRead(tid=24, to_read={3:b952e66c:::smithi12138290-29:head=0,118784,0}, subchunks={3:b952e66c:::smithi12138290-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.582633+0000", 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.425833423, 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043765009999999997, 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.042 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.582633+0000", 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.582633+0000", 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.582633+0000", 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3500000000000002e-07 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.582639+0000", 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0179999999999998e-06 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.582649+0000", 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0166000000000001e-05 2024-09-16T03:40:37.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.582661+0000", 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1919000000000001e-05 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.586922+0000", 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042604089999999997 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.587009+0000", 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7553999999999996e-05 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:264 3.7s0 3:e06e137e:::smithi12138290-27:head [copy-from ver 4] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.417789736, 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012313443, 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.046 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 264 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590679+0000", 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1739999999999999e-06 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590686+0000", 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.43e-06 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590709+0000", 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2748999999999999e-05 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590718+0000", 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0359999999999995e-06 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590860+0000", 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000142208 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590966+0000", 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000106439 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.599409+0000", 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008442583 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600331+0000", 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000922202 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.602907+0000", 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002575787 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.602990+0000", 2024-09-16T03:40:37.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2835000000000002e-05 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:265 3.7s0 3:e06e137e:::smithi12138290-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.417667896999999, 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0092800720000000003, 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:37.052 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 265 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:40:37.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590799+0000", 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590804+0000", 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3009999999999997e-06 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590832+0000", 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7851000000000002e-05 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590849+0000", 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6691e-05 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.591224+0000", 2024-09-16T03:40:37.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037502100000000001 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.591256+0000", 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.184e-05 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.599957+0000", 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087011659999999998 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600004+0000", 2024-09-16T03:40:37.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7218000000000001e-05 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600078+0000", 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4065000000000003e-05 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:296 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:40:37.057 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:40:37.058 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.537347579, 2024-09-16T03:40:37.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084845249999999997, 2024-09-16T03:40:37.058 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.058 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.058 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:37.058 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:37.058 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:37.058 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.058 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 296 2024-09-16T03:40:37.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471120+0000", 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.804e-06 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471128+0000", 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0199999999999994e-06 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471144+0000", 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5957999999999999e-05 2024-09-16T03:40:37.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471153+0000", 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7299999999999994e-06 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471238+0000", 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4860999999999999e-05 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:37.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471244+0000", 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0859999999999998e-06 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.473493+0000", 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022493750000000001 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.473624+0000", 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130891 2024-09-16T03:40:37.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.479603+0000", 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059788000000000003 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:297 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:40:37.063 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.537291178, 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094313820000000003, 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 297 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9900000000000001e-07 2024-09-16T03:40:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471180+0000", 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4619999999999998e-06 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471186+0000", 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3130000000000004e-06 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471190+0000", 2024-09-16T03:40:37.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.194e-06 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471314+0000", 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012333299999999999 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471322+0000", 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8280000000000003e-06 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.476742+0000", 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054202740000000001 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.476849+0000", 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010663899999999999 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480606+0000", 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037578400000000001 2024-09-16T03:40:37.068 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:298 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.537242651, 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011872347, 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:37.069 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 298 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:40:37.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2399999999999999e-07 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471227+0000", 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1149999999999998e-06 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471232+0000", 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.152e-06 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471237+0000", 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9540000000000003e-06 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471356+0000", 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011896900000000001 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:37.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471364+0000", 2024-09-16T03:40:37.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:40:37.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480706+0000", 2024-09-16T03:40:37.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093421570000000002 2024-09-16T03:40:37.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480820+0000", 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011411100000000001 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.483096+0000", 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022753819999999998 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:308 3.7s0 3:e441e0cd:::smithi12138290-34:head [create,setxattr h^blbcaNtSmmsQtZPdaRtPkhOOgN_Vi\\f`NtsZ[V (561) in=40b,setxattr T (17332) in=1b,setxattr UVoaZteYfmpthpcfl[gebaR\\TdqcNkhPRRdrgsX] (279) in=40b,setxattr bTPgfXb\\YXtXkQag\\Qh`hlXr[posd]]cje]^eebl (1121) in=40b,setxattr hm^gakWSOOOfNOTph^tma]iaNsaqfjqZYb\\seZNW (3297) in=40b,setxattr YiNn[]m_q`YlQlaV^mRn^ (17332) in=21b,setxattr \\tllpkXThmWXSkOopogrdtjY]f^a^sem_krafjfi (947) in=40b,setxattr RXiRZRQVSaUTaZmQT^bfqZhS^mZsqadaPdW]ijm` (1963) in=40b,setxattr WTmsTZhNpU`tZY\\^aROaNdmtYsjjsgWWboehPOP` (19508) in=40b,setxattr UZmWZeWaoiX]jmbn][UadnNkcZc_hdcjidtNsRYo (1999) in=40b,setxattr NSebjRYYQ`SbaatVjZWNab`cenkljRq`^]Xr\\jRm (11788) in=40b,setxattr ip`daUaop[rdbNPo\\^c[Wcelsl]a][RScQZ[kfa^ (18524) in=40b,setxattr qU_qcNcQpNkegs_Od`jieTXX[mto]lsR^jTcYprR (593) in=40b,setxattr dboVg`srq[bReagdR]fX\\Pe`Warrp]\\[XpSipTN[ (1683) in=40b,setxattr ZdTaVj[ScjsOiQ]a_]R[RQWoYohnjd`fi[dYmfs` (840) in=40b,setxattr N_[blmbpstWhtUgW]ZjZ[\\O_erqcnQN]R_V^U\\RT (1015) in=40b,setxattr jXbP_mNsadnq`UVfmc^lfZUs\\r_[ZnrahqTUk^Zp (383) in=40b,setxattr nmemd\\ipehj\\q^UVY_YtSrhZjcabNXhlai_noQSO (807) in=40b,setxattr Rbc\\cahhlcrckUioieUbQnrseNkZkdPf\\rUmNb]S (384) in=40b,setxattr ZiOkXqg^PobTdZgrtZ]]l^Rap_tfSQoUQqaefj]] (69) in=40b] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.5179817230000001, 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013488215, 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 308 2024-09-16T03:40:37.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490486+0000", 2024-09-16T03:40:37.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.601e-06 2024-09-16T03:40:37.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490741+0000", 2024-09-16T03:40:37.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00025543300000000001 2024-09-16T03:40:37.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490756+0000", 2024-09-16T03:40:37.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4949e-05 2024-09-16T03:40:37.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490764+0000", 2024-09-16T03:40:37.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1659999999999997e-06 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.492684+0000", 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019205509999999999 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.493193+0000", 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050880499999999998 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.494675+0000", 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001482443 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.496347+0000", 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001671358 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.503822+0000", 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074755639999999996 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.503973+0000", 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000150345 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.0s1 52/31 ECSubWrite(tid=25, reqid=client.4615.0:334, at_version=52'14, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.481008+0000", 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.5274579270000004, 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053169339999999997, 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.083 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481008+0000", 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481008+0000", 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481009+0000", 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9100000000000003e-07 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481064+0000", 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4620000000000002e-05 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481075+0000", 2024-09-16T03:40:37.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1053000000000001e-05 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481082+0000", 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4560000000000003e-06 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481131+0000", 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9168000000000001e-05 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481137+0000", 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8760000000000002e-06 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.486292+0000", 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051551729999999999 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.486325+0000", 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2897e-05 2024-09-16T03:40:37.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.0s1 52/31 ECSubWrite(tid=26, reqid=client.4615.0:335, at_version=52'15, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.481900+0000", 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.5265656810000001, 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044494519999999996, 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481900+0000", 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481900+0000", 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481901+0000", 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6199999999999997e-07 2024-09-16T03:40:37.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481917+0000", 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5811000000000002e-05 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481930+0000", 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3497e-05 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481939+0000", 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0940000000000008e-06 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481989+0000", 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0003999999999998e-05 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481994+0000", 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8359999999999997e-06 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.486329+0000", 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043346069999999999 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.486350+0000", 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0941000000000002e-05 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.ds2 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.505108+0000", 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.5033580940000002, 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046943510000000003, 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505108+0000", 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505108+0000", 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505109+0000", 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0240000000000001e-06 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505132+0000", 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2670000000000001e-05 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505151+0000", 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8975e-05 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505158+0000", 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3000000000000004e-06 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505216+0000", 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8352e-05 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505240+0000", 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3754e-05 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.509755+0000", 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045143680000000004 2024-09-16T03:40:37.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.509802+0000", 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7908000000000003e-05 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.9s3 53/31 ECSubWrite(tid=44, reqid=client.4615.0:364, at_version=53'23, trim_to=0'0, roll_forward_to=53'20))", 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.520717+0000", 2024-09-16T03:40:37.097 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.4877492229999998, 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050875099999999999, 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520717+0000", 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520717+0000", 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520727+0000", 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6039999999999999e-06 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520779+0000", 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2864000000000001e-05 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520793+0000", 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3585e-05 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520804+0000", 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1498000000000001e-05 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520897+0000", 2024-09-16T03:40:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2527000000000004e-05 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520913+0000", 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6222000000000002e-05 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.525776+0000", 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048629440000000001 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.525804+0000", 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8266e-05 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s4 54/31 ECSubWrite(tid=56, reqid=osd.6.0:56, at_version=54'45, trim_to=0'0, roll_forward_to=53'43))", 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.510581+0000", 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.4978853340000002, 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051499390000000001, 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510581+0000", 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510581+0000", 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510582+0000", 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9600000000000008e-07 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510622+0000", 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0048e-05 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510642+0000", 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9984999999999999e-05 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510652+0000", 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0068000000000001e-05 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510846+0000", 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000193573 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510857+0000", 2024-09-16T03:40:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1163e-05 2024-09-16T03:40:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515690+0000", 2024-09-16T03:40:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048328490000000002 2024-09-16T03:40:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515731+0000", 2024-09-16T03:40:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1257000000000002e-05 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.ds2 54/31 ECSubWrite(tid=37, reqid=osd.12.0:37, at_version=54'19, trim_to=0'0, roll_forward_to=53'15))", 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.510873+0000", 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.4975931980000001, 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048256439999999996, 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510873+0000", 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510873+0000", 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510874+0000", 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7000000000000003e-07 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510897+0000", 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3433000000000001e-05 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510912+0000", 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4245e-05 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510921+0000", 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.166e-06 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.511029+0000", 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010799299999999999 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.511038+0000", 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1079999999999999e-06 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515663+0000", 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046254470000000004 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515699+0000", 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5382000000000001e-05 2024-09-16T03:40:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s4 54/31 ECSubWrite(tid=57, reqid=osd.6.0:56, at_version=0'0, trim_to=0'0, roll_forward_to=54'45))", 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.518458+0000", 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.4900079640000001, 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005602709, 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518458+0000", 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518458+0000", 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518459+0000", 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518471+0000", 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1647999999999999e-05 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518484+0000", 2024-09-16T03:40:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3621000000000001e-05 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518494+0000", 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7399999999999999e-06 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518574+0000", 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9575999999999998e-05 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518584+0000", 2024-09-16T03:40:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0359e-05 2024-09-16T03:40:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.524019+0000", 2024-09-16T03:40:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005435151 2024-09-16T03:40:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.524061+0000", 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1631000000000002e-05 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.0s1 54/31 ECSubWrite(tid=30, reqid=client.4615.0:421, at_version=0'0, trim_to=0'0, roll_forward_to=54'16))", 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.518829+0000", 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.4896369819999999, 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052386389999999998, 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518829+0000", 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518829+0000", 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518830+0000", 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5900000000000005e-07 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518840+0000", 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0241000000000001e-05 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518852+0000", 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1989e-05 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518866+0000", 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3515e-05 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518938+0000", 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2094000000000005e-05 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518949+0000", 2024-09-16T03:40:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0852999999999999e-05 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.524037+0000", 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050878080000000001 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.524068+0000", 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1180000000000003e-05 2024-09-16T03:40:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:37.122 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:37.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:40:37.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/journal --op export --pgid 3.fs1 --file /var/log/ceph/exp.3.fs1.13 2024-09-16T03:40:37.125 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:37.122+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:37.224 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:37.222+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:37.241 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:37.241 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:37.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:40:37.324 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:37.323+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:37.394 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:37.395 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:37.395 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:37.395 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:37.395 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:37.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:40:37.426 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:37.424+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:37.472 INFO:tasks.rados.rados.0.smithi121.stdout:172: done (1 left) 2024-09-16T03:40:37.472 INFO:tasks.rados.rados.0.smithi121.stdout:173: done (0 left) 2024-09-16T03:40:37.472 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 16 current snap is 15 2024-09-16T03:40:37.473 INFO:tasks.rados.rados.0.smithi121.stdout:174: seq_num 96 ranges {262144=458752,720896=688128,1409024=475136} 2024-09-16T03:40:37.476 INFO:tasks.rados.rados.0.smithi121.stdout:174: writing smithi12138290-16 from 262144 to 720896 tid 1 2024-09-16T03:40:37.479 INFO:tasks.rados.rados.0.smithi121.stdout:174: writing smithi12138290-16 from 720896 to 1409024 tid 2 2024-09-16T03:40:37.483 INFO:tasks.rados.rados.0.smithi121.stdout:174: writing smithi12138290-16 from 1409024 to 1884160 tid 3 2024-09-16T03:40:37.483 INFO:tasks.rados.rados.0.smithi121.stdout:175: snap_create 2024-09-16T03:40:37.483 INFO:tasks.rados.rados.0.smithi121.stdout:174: finishing write tid 1 to smithi12138290-16 2024-09-16T03:40:37.485 INFO:tasks.rados.rados.0.smithi121.stdout:174: finishing write tid 2 to smithi12138290-16 2024-09-16T03:40:37.487 INFO:tasks.rados.rados.0.smithi121.stdout:174: finishing write tid 3 to smithi12138290-16 2024-09-16T03:40:37.490 INFO:tasks.rados.rados.0.smithi121.stdout:174: finishing write tid 4 to smithi12138290-16 2024-09-16T03:40:37.491 INFO:tasks.rados.rados.0.smithi121.stdout:174: finishing write tid 5 to smithi12138290-16 2024-09-16T03:40:37.491 INFO:tasks.rados.rados.0.smithi121.stdout:174: oid 16 updating version 0 to 23 2024-09-16T03:40:37.491 INFO:tasks.rados.rados.0.smithi121.stdout:174: oid 16 version 23 is already newer than 21 2024-09-16T03:40:37.492 INFO:tasks.rados.rados.0.smithi121.stdout:174: oid 16 version 23 is already newer than 20 2024-09-16T03:40:37.492 INFO:tasks.rados.rados.0.smithi121.stdout:174: oid 16 version 23 is already newer than 22 2024-09-16T03:40:37.492 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 16 v 23 (ObjNum 96 snap 15 seq_num 96) dirty exists 2024-09-16T03:40:37.492 INFO:tasks.rados.rados.0.smithi121.stdout:174: left oid 16 (ObjNum 96 snap 15 seq_num 96) 2024-09-16T03:40:37.511 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:37.511 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:37.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:37.511 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:37.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.511 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=27, reqid=client.4615.0:219, at_version=37'9, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:37.511 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.557379+0000", 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 20.951153038000001, 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002658929, 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557379+0000", 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557379+0000", 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557380+0000", 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6100000000000001e-07 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557395+0000", 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5763000000000001e-05 2024-09-16T03:40:37.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557408+0000", 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2183e-05 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557415+0000", 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.622e-06 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557481+0000", 2024-09-16T03:40:37.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5617999999999994e-05 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557490+0000", 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0280000000000006e-06 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.560006+0000", 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025159779999999999 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.560038+0000", 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2175999999999998e-05 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=30, reqid=client.4615.0:314, at_version=49'20, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.0149614769999999, 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0049994970000000003, 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.516 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6100000000000002e-07 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493601+0000", 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9561999999999999e-05 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493614+0000", 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3421000000000001e-05 2024-09-16T03:40:37.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493623+0000", 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4050000000000006e-06 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493705+0000", 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1500000000000002e-05 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493719+0000", 2024-09-16T03:40:37.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3866000000000001e-05 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.498525+0000", 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048063230000000004 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.498570+0000", 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5058999999999997e-05 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.520 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=31, reqid=osd.8.0:31, at_version=49'22, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.0130746169999991, 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059536579999999997, 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.521 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495459+0000", 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3799999999999996e-07 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495484+0000", 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5601000000000001e-05 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495505+0000", 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0460999999999999e-05 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495514+0000", 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1390000000000001e-06 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495592+0000", 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7877e-05 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495607+0000", 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5576000000000001e-05 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501362+0000", 2024-09-16T03:40:37.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057546280000000003 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501411+0000", 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9438000000000003e-05 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 49/31 ECSubWrite(tid=84, reqid=osd.0.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=49'20))", 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.497586+0000", 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.0109465069999999, 2024-09-16T03:40:37.525 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041019860000000002, 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497586+0000", 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497586+0000", 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497587+0000", 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0100000000000001e-06 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497598+0000", 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1365999999999999e-05 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497613+0000", 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4752e-05 2024-09-16T03:40:37.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497619+0000", 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5980000000000002e-06 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497688+0000", 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8973999999999994e-05 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.528 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497698+0000", 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5550000000000008e-06 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501634+0000", 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039355780000000003 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501688+0000", 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4153000000000001e-05 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.529 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 52/31 ECSubWrite(tid=86, reqid=client.4615.0:345, at_version=52'22, trim_to=0'0, roll_forward_to=49'20))", 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.506376+0000", 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.0021562959999999, 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042306510000000002, 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506376+0000", 2024-09-16T03:40:37.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506376+0000", 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506376+0000", 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4999999999999999e-07 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.531 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506480+0000", 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000104156 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506490+0000", 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6860000000000001e-06 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506498+0000", 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1790000000000008e-06 2024-09-16T03:40:37.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506549+0000", 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1214000000000001e-05 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506556+0000", 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0149999999999998e-06 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.510575+0000", 2024-09-16T03:40:37.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004018532 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.510607+0000", 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1718999999999999e-05 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 53/31 ECSubWrite(tid=89, reqid=osd.0.0:88, at_version=0'0, trim_to=0'0, roll_forward_to=53'24))", 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.500220+0000", 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.0083124320000003, 2024-09-16T03:40:37.534 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040807619999999999, 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500220+0000", 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500220+0000", 2024-09-16T03:40:37.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500220+0000", 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9999999999999997e-07 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500232+0000", 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1488000000000001e-05 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500240+0000", 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2979999999999992e-06 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500246+0000", 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5929999999999996e-06 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500297+0000", 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0886e-05 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500308+0000", 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1080999999999999e-05 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504258+0000", 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039503000000000003 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504301+0000", 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2515999999999997e-05 2024-09-16T03:40:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=27, reqid=client.4615.0:354, at_version=53'8, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.503722+0000", 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.004810054, 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003563376, 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.539 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.503722+0000", 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.503722+0000", 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.503722+0000", 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1899999999999999e-07 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.503748+0000", 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6021999999999999e-05 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.503760+0000", 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1474000000000001e-05 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.503769+0000", 2024-09-16T03:40:37.541 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0450000000000006e-06 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.503837+0000", 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7782999999999998e-05 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.503847+0000", 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9850000000000001e-06 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.542 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507253+0000", 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00340595 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507286+0000", 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2997999999999998e-05 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.543 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.0037851800000004, 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068498659999999996, 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:40:37.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504748+0000", 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4499999999999997e-07 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504768+0000", 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0582999999999999e-05 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504782+0000", 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3883000000000001e-05 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504790+0000", 2024-09-16T03:40:37.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3389999999999993e-06 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504921+0000", 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013093800000000001 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504930+0000", 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7499999999999992e-06 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.547 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511566+0000", 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066358449999999996 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511597+0000", 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0883000000000002e-05 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=29, reqid=client.4615.0:356, at_version=53'10, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:37.548 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.506019+0000", 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.0025130000000004, 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0056103890000000003, 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506019+0000", 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.549 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506019+0000", 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506020+0000", 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506032+0000", 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2446000000000001e-05 2024-09-16T03:40:37.550 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506042+0000", 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6579999999999997e-06 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506047+0000", 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5199999999999997e-06 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506099+0000", 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1518999999999999e-05 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506110+0000", 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0496000000000001e-05 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511600+0000", 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054906190000000004 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.552 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511630+0000", 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9447000000000001e-05 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 53/31 ECSubWrite(tid=43, reqid=osd.8.0:43, at_version=53'32, trim_to=0'0, roll_forward_to=51'30))", 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.507530+0000", 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.0010022760000004, 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041212599999999999, 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.553 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507530+0000", 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507530+0000", 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.554 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507530+0000", 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8300000000000001e-07 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507552+0000", 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1622000000000001e-05 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507565+0000", 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3135999999999999e-05 2024-09-16T03:40:37.555 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507573+0000", 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2719999999999997e-06 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507629+0000", 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6014000000000003e-05 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507638+0000", 2024-09-16T03:40:37.556 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2939999999999994e-06 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511632+0000", 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039935769999999999 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511651+0000", 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9162000000000001e-05 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.557 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=30, reqid=client.4615.0:357, at_version=53'12, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.508274+0000", 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.0002580490000001, 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0056031689999999999, 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.558 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508274+0000", 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508274+0000", 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508275+0000", 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2500000000000005e-07 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.559 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508534+0000", 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025936500000000001 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508549+0000", 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4664000000000001e-05 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508557+0000", 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4079999999999998e-06 2024-09-16T03:40:37.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508619+0000", 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1467e-05 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508628+0000", 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2210000000000003e-06 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513823+0000", 2024-09-16T03:40:37.561 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051947820000000002 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513877+0000", 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4537000000000003e-05 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 53/31 ECSubWrite(tid=20, reqid=client.4615.0:376, at_version=53'13, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.544496+0000", 2024-09-16T03:40:37.562 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9640357369999997, 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059495989999999999, 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544496+0000", 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.563 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544496+0000", 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544498+0000", 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3519999999999999e-06 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545106+0000", 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00060862000000000004 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.564 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.565 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.565 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545125+0000", 2024-09-16T03:40:37.565 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9052999999999999e-05 2024-09-16T03:40:37.565 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.565 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.565 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.565 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545136+0000", 2024-09-16T03:40:37.568 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0125e-05 2024-09-16T03:40:37.568 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.568 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.568 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.568 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545201+0000", 2024-09-16T03:40:37.568 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5769e-05 2024-09-16T03:40:37.568 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545212+0000", 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0611000000000001e-05 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550406+0000", 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051939519999999999 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550446+0000", 2024-09-16T03:40:37.569 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0117000000000002e-05 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 53/31 ECSubWrite(tid=21, reqid=client.4615.0:377, at_version=53'14, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.545414+0000", 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9631181479999995, 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050486039999999999, 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.570 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545414+0000", 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545414+0000", 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545415+0000", 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1699999999999997e-07 2024-09-16T03:40:37.571 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.572 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.572 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.572 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545439+0000", 2024-09-16T03:40:37.572 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4539999999999999e-05 2024-09-16T03:40:37.572 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.572 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.572 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.572 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545452+0000", 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2150000000000001e-05 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545460+0000", 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1929999999999993e-06 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545704+0000", 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000243985 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.573 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545714+0000", 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0103e-05 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550450+0000", 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047365330000000002 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550463+0000", 2024-09-16T03:40:37.574 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2283000000000001e-05 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 53/31 ECSubWrite(tid=93, reqid=client.4615.0:389, at_version=53'28, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.554348+0000", 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9541844929999996, 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047364149999999999, 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.575 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554348+0000", 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554348+0000", 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554349+0000", 2024-09-16T03:40:37.576 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.16e-07 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554719+0000", 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000370892 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554732+0000", 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2918999999999999e-05 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.577 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554741+0000", 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4230000000000005e-06 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554821+0000", 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9753999999999996e-05 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554830+0000", 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1630000000000002e-06 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.578 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.559049+0000", 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042190839999999997 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.559084+0000", 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5264e-05 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.579 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 53/31 ECSubWrite(tid=97, reqid=client.4615.0:390, at_version=53'29, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.555896+0000", 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9526363020000002, 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004138236, 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.555896+0000", 2024-09-16T03:40:37.580 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.555896+0000", 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.555897+0000", 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7100000000000001e-07 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.581 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.555956+0000", 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9116000000000001e-05 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.555968+0000", 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2248e-05 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.555975+0000", 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3490000000000003e-06 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.582 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.556044+0000", 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8516000000000006e-05 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.556054+0000", 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0628999999999999e-05 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.560009+0000", 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039542140000000002 2024-09-16T03:40:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.560034+0000", 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5493000000000001e-05 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 53/31 ECSubWrite(tid=98, reqid=client.4615.0:391, at_version=53'30, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.556938+0000", 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9515937299999999, 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031241709999999998, 2024-09-16T03:40:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.556938+0000", 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.556938+0000", 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.585 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.556939+0000", 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2600000000000002e-07 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.556950+0000", 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1238e-05 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.556960+0000", 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5729999999999997e-06 2024-09-16T03:40:37.586 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.556967+0000", 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3170000000000004e-06 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.557022+0000", 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4914999999999999e-05 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.587 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.557033+0000", 2024-09-16T03:40:37.588 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1110999999999999e-05 2024-09-16T03:40:37.588 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.588 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.588 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.588 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.560039+0000", 2024-09-16T03:40:37.588 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003005962 2024-09-16T03:40:37.588 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.588 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.588 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.588 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.560063+0000", 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3428999999999999e-05 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 54/31 ECSubWrite(tid=36, reqid=osd.12.0:36, at_version=54'17, trim_to=0'0, roll_forward_to=53'15))", 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:33.510878+0000", 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.9976542940000002, 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0035059010000000001, 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.589 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.510878+0000", 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.510878+0000", 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.590 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.510878+0000", 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7099999999999998e-07 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.510914+0000", 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5479999999999999e-05 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.510926+0000", 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2198000000000001e-05 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.591 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.510937+0000", 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1341e-05 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.511005+0000", 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7936999999999997e-05 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.511018+0000", 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3175e-05 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.592 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.514315+0000", 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003297038 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.514384+0000", 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8460999999999994e-05 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.593 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 54/31 ECSubWrite(tid=37, reqid=osd.12.0:37, at_version=54'19, trim_to=0'0, roll_forward_to=53'15))", 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:33.510960+0000", 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.9975717039999998, 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00385185, 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.510960+0000", 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.594 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.510960+0000", 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.510961+0000", 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0200000000000003e-07 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.595 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.510994+0000", 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3596000000000002e-05 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.511004+0000", 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9729999999999992e-06 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.511012+0000", 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.695e-06 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.596 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.511342+0000", 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00032988599999999999 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.511351+0000", 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0480000000000004e-06 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.514777+0000", 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034261439999999999 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.597 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.514812+0000", 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5105999999999999e-05 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 54/31 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=54'16, trim_to=0'0, roll_forward_to=53'14))", 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:33.515283+0000", 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.9932493650000001, 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002717798, 2024-09-16T03:40:37.598 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.515283+0000", 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.515283+0000", 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.515283+0000", 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.35e-07 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.515309+0000", 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6363000000000002e-05 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.515321+0000", 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1922e-05 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.515330+0000", 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7679999999999997e-06 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.515402+0000", 2024-09-16T03:40:37.601 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1499000000000001e-05 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.515413+0000", 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0939e-05 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.517954+0000", 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025416499999999999 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.602 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.518001+0000", 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6421999999999997e-05 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 54/31 ECSubWrite(tid=38, reqid=osd.12.0:37, at_version=0'0, trim_to=0'0, roll_forward_to=54'19))", 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:33.516293+0000", 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.9922395769999999, 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004306069, 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.603 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.516293+0000", 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.516293+0000", 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.604 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.516294+0000", 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6700000000000002e-07 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.516309+0000", 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5322999999999999e-05 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.516322+0000", 2024-09-16T03:40:37.605 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.296e-05 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.516331+0000", 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4020000000000008e-06 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.516399+0000", 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7565999999999997e-05 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.516409+0000", 2024-09-16T03:40:37.606 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9960000000000005e-06 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.520563+0000", 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004153988 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:33.520599+0000", 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5867000000000001e-05 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.607 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.608 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.608 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:37.608 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:40:37.610 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:37.607+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:37.610 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:40:37.710 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:37.708+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:37.714 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:37.714 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:37.714 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:37.714 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:37.724 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:40:37.811 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:37.810+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:37.826 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:37.826 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:37.826 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:37.826 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:37.826 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:37.834 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:40:37.913 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:37.911+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:37.938 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:37.938 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:37.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:37.938 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:37.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.938 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:268 3.1s0 3:82a78c8d:::smithi12138290-1:head [append 0~196608] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:37.938 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.627532+0000", 2024-09-16T03:40:37.938 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 16.307767364, 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075158530000000003, 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 268 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627532+0000", 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627532+0000", 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627533+0000", 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6599999999999995e-07 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.941 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.941 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627834+0000", 2024-09-16T03:40:37.941 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00030093699999999998 2024-09-16T03:40:37.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.941 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.941 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.941 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627844+0000", 2024-09-16T03:40:37.941 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0478e-05 2024-09-16T03:40:37.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.941 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627850+0000", 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7910000000000001e-06 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.628455+0000", 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00060544400000000001 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.628522+0000", 2024-09-16T03:40:37.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6670999999999999e-05 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.629026+0000", 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050412799999999995 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630252+0000", 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001225711 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.634935+0000", 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046835130000000003 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635048+0000", 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000112414 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.944 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:269 3.1s0 3:82a78c8d:::smithi12138290-1:head [setxattr _header (55) in=7b] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.627861+0000", 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 16.307438661999999, 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073128849999999999, 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 269 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.945 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627861+0000", 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627861+0000", 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.946 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627861+0000", 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8100000000000003e-07 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627864+0000", 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8490000000000002e-06 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627870+0000", 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0299999999999999e-06 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627873+0000", 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2440000000000002e-06 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.629522+0000", 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001648634 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.629585+0000", 2024-09-16T03:40:37.948 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2908000000000001e-05 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.629951+0000", 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00036612199999999998 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630681+0000", 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073026799999999995 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.949 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635120+0000", 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044385550000000003 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635174+0000", 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3693999999999998e-05 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:270 3.1s0 3:82a78c8d:::smithi12138290-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:37.950 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:37.951 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 16.307421575999999, 2024-09-16T03:40:37.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086784840000000002, 2024-09-16T03:40:37.951 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.951 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.951 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:37.951 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:37.951 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:37.951 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 270 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8299999999999998e-07 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627880+0000", 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.716e-06 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627895+0000", 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4904e-05 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627900+0000", 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4990000000000002e-06 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630550+0000", 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0026501960000000001 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630645+0000", 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4790999999999994e-05 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635427+0000", 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047821740000000001 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635478+0000", 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0603000000000003e-05 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.636556+0000", 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001078318 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:280 3.1s0 3:8df00818:::smithi12138290-14:head [copy-from ver 2] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.569179+0000", 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.366120465, 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080900399999999997, 2024-09-16T03:40:37.956 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 280 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569179+0000", 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569179+0000", 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569181+0000", 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3970000000000001e-06 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569193+0000", 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1744e-05 2024-09-16T03:40:37.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569210+0000", 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7286e-05 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569223+0000", 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2852e-05 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569304+0000", 2024-09-16T03:40:37.959 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0815000000000005e-05 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569310+0000", 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9800000000000003e-06 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571309+0000", 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001999158 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.960 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571423+0000", 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011391399999999999 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575663+0000", 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042401619999999996 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.961 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.576691+0000", 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010281489999999999 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577144+0000", 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00045261500000000002 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577269+0000", 2024-09-16T03:40:37.962 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012496799999999999 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:281 3.1s0 3:8df00818:::smithi12138290-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.569246+0000", 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.366053178, 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073290630000000002, 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:37.963 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:37.964 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:37.964 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.964 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 281 2024-09-16T03:40:37.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.964 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569246+0000", 2024-09-16T03:40:37.964 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569246+0000", 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569247+0000", 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2600000000000002e-07 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569251+0000", 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3320000000000002e-06 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569264+0000", 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2839e-05 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569270+0000", 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8039999999999998e-06 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569378+0000", 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010869700000000001 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569385+0000", 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8569999999999997e-06 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571608+0000", 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022225389999999999 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571658+0000", 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9840999999999999e-05 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.576351+0000", 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046931289999999999 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.576453+0000", 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000101808 2024-09-16T03:40:37.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.576575+0000", 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012259099999999999 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:301 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~491520] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:40:37.969 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.4524412309999999, 2024-09-16T03:40:37.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013883662999999999, 2024-09-16T03:40:37.970 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.970 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.970 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.970 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:37.970 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:37.970 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.970 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 301 2024-09-16T03:40:37.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482860+0000", 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.911e-06 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483803+0000", 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00094328700000000001 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.972 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483820+0000", 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6249e-05 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483828+0000", 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.704e-06 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483895+0000", 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6839000000000002e-05 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483902+0000", 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.485797+0000", 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018946049999999999 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.485916+0000", 2024-09-16T03:40:37.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011867 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487208+0000", 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012921110000000001 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493186+0000", 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059786010000000001 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496639+0000", 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034530749999999999 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496742+0000", 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102394 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.976 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:302 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~770048] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.448161958, 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014729638999999999, 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 302 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.977 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.978 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487139+0000", 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.378e-06 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488116+0000", 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00097674700000000003 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488127+0000", 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1817999999999999e-05 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488135+0000", 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6620000000000005e-06 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489443+0000", 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001308303 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489564+0000", 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000120664 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490525+0000", 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00096115100000000004 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493256+0000", 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027313429999999998 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501784+0000", 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0085277840000000001 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501867+0000", 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2788999999999996e-05 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.982 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:303 3.1s0 3:8a735ecd:::smithi12138290-30:head [setxattr _header (57) in=7b] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.447150422, 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013931318, 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 303 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.983 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488150+0000", 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000001e-07 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488152+0000", 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.509e-06 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488157+0000", 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5739999999999997e-06 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488160+0000", 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.349e-06 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491536+0000", 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033756799999999998 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491638+0000", 2024-09-16T03:40:37.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010186799999999999 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492139+0000", 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050129999999999999 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493257+0000", 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001118551 2024-09-16T03:40:37.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501989+0000", 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087319819999999992 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502080+0000", 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0827000000000003e-05 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:304 3.1s0 3:8a735ecd:::smithi12138290-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.4471334440000003, 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.024338029000000001, 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:37.989 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 304 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.53e-07 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488169+0000", 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3309999999999998e-06 2024-09-16T03:40:37.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488173+0000", 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488176+0000", 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0539999999999999e-06 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492435+0000", 2024-09-16T03:40:37.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042594399999999998 2024-09-16T03:40:37.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:37.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492534+0000", 2024-09-16T03:40:37.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8558000000000001e-05 2024-09-16T03:40:37.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505269+0000", 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012735076 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505321+0000", 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1977000000000001e-05 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.512504+0000", 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0071833060000000004 2024-09-16T03:40:37.994 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:309 3.1s0 3:82a78c8d:::smithi12138290-1:head [copy-from ver 4] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.4444739910000006, 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020962743999999998, 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:37.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 309 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:37.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490827+0000", 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490832+0000", 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4550000000000003e-06 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490845+0000", 2024-09-16T03:40:37.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2951e-05 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490873+0000", 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7566999999999999e-05 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492571+0000", 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001698359 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:37.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492650+0000", 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8858999999999998e-05 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503971+0000", 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011320805 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511333+0000", 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073624160000000001 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:37.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511735+0000", 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00040226999999999998 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511788+0000", 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2757999999999999e-05 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:310 3.1s0 3:82a78c8d:::smithi12138290-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.4444080120000002, 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014337858, 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 310 2024-09-16T03:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490892+0000", 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.37e-07 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490896+0000", 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3710000000000002e-06 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490932+0000", 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5796000000000001e-05 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490941+0000", 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8589999999999994e-06 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493109+0000", 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021679310000000001 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:38.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493167+0000", 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7572999999999999e-05 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505077+0000", 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011909968999999999 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505164+0000", 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7409000000000003e-05 2024-09-16T03:40:38.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505229+0000", 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5213000000000001e-05 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s1 49/31 ECSubWrite(tid=30, reqid=client.4615.0:314, at_version=49'20, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.493624+0000", 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.4416754760000003, 2024-09-16T03:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0074256740000000002, 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493624+0000", 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493624+0000", 2024-09-16T03:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493624+0000", 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3099999999999998e-07 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493650+0000", 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5216e-05 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493660+0000", 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0088999999999999e-05 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493669+0000", 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1139999999999995e-06 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493736+0000", 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6883000000000003e-05 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493745+0000", 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5540000000000003e-06 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501011+0000", 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00726627 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501050+0000", 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8117000000000001e-05 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=22, reqid=client.4615.0:308, at_version=49'14, trim_to=0'0, roll_forward_to=42'12))", 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.494598+0000", 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.4407012619999993, 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084633480000000007, 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.011 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494598+0000", 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494598+0000", 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494599+0000", 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6899999999999998e-07 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494838+0000", 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000239409 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494848+0000", 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8859999999999999e-06 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494855+0000", 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9739999999999996e-06 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494918+0000", 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3478000000000004e-05 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494925+0000", 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1940000000000003e-06 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503029+0000", 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081040469999999996 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503061+0000", 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2891000000000001e-05 2024-09-16T03:40:38.015 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 49/31 ECSubWriteReply(tid=34, last_complete=49'24, committed=1, applied=1))", 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.503107+0000", 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.4321926059999992, 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081770950000000005, 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503107+0000", 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503107+0000", 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503108+0000", 2024-09-16T03:40:38.017 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.61e-07 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503112+0000", 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9239999999999998e-06 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503122+0000", 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4599999999999992e-06 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503129+0000", 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6540000000000002e-06 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511256+0000", 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081276979999999992 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511284+0000", 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7597999999999999e-05 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 49/31 ECSubWriteReply(tid=34, last_complete=49'24, committed=1, applied=1))", 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.503421+0000", 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.4318786049999996, 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078850580000000003, 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503421+0000", 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503421+0000", 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503422+0000", 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503426+0000", 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4689999999999999e-06 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503439+0000", 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2802e-05 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503447+0000", 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0669999999999992e-06 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511292+0000", 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078452020000000008 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511306+0000", 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3689000000000001e-05 2024-09-16T03:40:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 49/31 ECSubWriteReply(tid=34, last_complete=49'24, committed=1, applied=1))", 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.4318256199999997, 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00786215, 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3099999999999998e-07 2024-09-16T03:40:38.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503477+0000", 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1520000000000001e-06 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503488+0000", 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0387000000000001e-05 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503494+0000", 2024-09-16T03:40:38.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9850000000000003e-06 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511323+0000", 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078294260000000004 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511336+0000", 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2669e-05 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.027 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:337 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~606208] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.492469+0000", 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.4428303979999999, 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0083634099999999999, 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.028 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 337 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.492469+0000", 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.492469+0000", 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.492470+0000", 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.511e-06 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493117+0000", 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00064688400000000002 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493129+0000", 2024-09-16T03:40:38.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.203e-05 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493135+0000", 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8139999999999997e-06 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493197+0000", 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2141e-05 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493305+0000", 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107426 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.494433+0000", 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001128471 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496206+0000", 2024-09-16T03:40:38.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001772608 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500752+0000", 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045464759999999998 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500832+0000", 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0049000000000006e-05 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.033 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:338 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~720896] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.496989+0000", 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.4383098810000003, 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0077073139999999998, 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.034 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 338 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496989+0000", 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496989+0000", 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496991+0000", 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0449999999999999e-06 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497710+0000", 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00071904 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497719+0000", 2024-09-16T03:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3409999999999992e-06 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497723+0000", 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4610000000000001e-06 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497761+0000", 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7469000000000003e-05 2024-09-16T03:40:38.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497832+0000", 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1612999999999994e-05 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.498416+0000", 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00058368299999999999 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.501013+0000", 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025971829999999999 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.504614+0000", 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036004779999999998 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.504697+0000", 2024-09-16T03:40:38.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3001e-05 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:341 3.1s0 3:8a735ecd:::smithi12138290-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.4375227410000004, 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.008644522, 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.040 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 341 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.041 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4400000000000001e-07 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497779+0000", 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7260000000000001e-06 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497783+0000", 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9509999999999999e-06 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497786+0000", 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8629999999999999e-06 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500932+0000", 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031462389999999999 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.501005+0000", 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3546000000000004e-05 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.505207+0000", 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042014330000000001 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.505269+0000", 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2337999999999998e-05 2024-09-16T03:40:38.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506421+0000", 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001152082 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds6 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.504801+0000", 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.4304980890000003, 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0082787090000000004, 2024-09-16T03:40:38.045 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504801+0000", 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504801+0000", 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504802+0000", 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1600000000000001e-07 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504821+0000", 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9117999999999999e-05 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504833+0000", 2024-09-16T03:40:38.047 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.151e-05 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504840+0000", 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5959999999999998e-06 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504899+0000", 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8693000000000002e-05 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504909+0000", 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6590000000000002e-06 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513048+0000", 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081393769999999997 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513080+0000", 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2039999999999998e-05 2024-09-16T03:40:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.050 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.050 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.050 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.050 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.050 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:38.051 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:40:38.052 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:38.049+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:38.153 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:38.151+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:38.155 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:38.155 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:38.155 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:38.155 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:38.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:40:38.254 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:38.252+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:38.307 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:38.307 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:38.307 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:38.307 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:38.307 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:38.315 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:40:38.355 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:38.353+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:38.420 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:38.420 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:38.420 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:38.420 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:38.420 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.420 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:213 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~688128] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:38.420 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.552584+0000", 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 21.864691665999999, 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059945609999999998, 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 213 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.421 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.552584+0000", 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.552584+0000", 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.552587+0000", 2024-09-16T03:40:38.422 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7300000000000001e-06 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.553514+0000", 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00092716800000000002 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.553527+0000", 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2991e-05 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.423 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.424 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.553533+0000", 2024-09-16T03:40:38.424 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7350000000000001e-06 2024-09-16T03:40:38.424 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.424 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.424 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.424 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.553604+0000", 2024-09-16T03:40:38.424 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1439999999999994e-05 2024-09-16T03:40:38.424 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.424 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.553667+0000", 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2627999999999997e-05 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.554786+0000", 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011192890000000001 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.425 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558234+0000", 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034485599999999998 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558488+0000", 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000253673 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558578+0000", 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0346999999999994e-05 2024-09-16T03:40:38.426 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:215 3.bs0 3:de0fee80:::smithi12138290-48:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.555429+0000", 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 21.861846645, 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005874036, 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.427 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 215 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555429+0000", 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.428 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555429+0000", 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555430+0000", 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9199999999999999e-07 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555434+0000", 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6029999999999998e-06 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.429 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555443+0000", 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8789999999999992e-06 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555450+0000", 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7109999999999998e-06 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557393+0000", 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019429829999999999 2024-09-16T03:40:38.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.557510+0000", 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011719599999999999 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558006+0000", 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00049595900000000003 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.560272+0000", 2024-09-16T03:40:38.431 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022660139999999998 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561233+0000", 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00096107099999999997 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561303+0000", 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9727999999999994e-05 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.432 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:216 3.bs0 3:de0fee80:::smithi12138290-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.555462+0000", 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 21.861813824999999, 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0071725290000000004, 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.433 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 216 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555462+0000", 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555462+0000", 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.434 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555462+0000", 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555465+0000", 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555473+0000", 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.909e-06 2024-09-16T03:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.555479+0000", 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8950000000000003e-06 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558776+0000", 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032970579999999998 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.558831+0000", 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5133999999999998e-05 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561771+0000", 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029393380000000001 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.561833+0000", 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2637000000000004e-05 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.437 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.562634+0000", 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080094400000000005 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:282 3.4s0 3:2307c0f5:::smithi12138290-28:head [append 0~458752] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.842484855, 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0093867299999999994, 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 282 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.439 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574793+0000", 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5799999999999999e-06 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575536+0000", 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00074309099999999998 2024-09-16T03:40:38.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575567+0000", 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0382000000000002e-05 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575590+0000", 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3215000000000001e-05 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.441 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575665+0000", 2024-09-16T03:40:38.442 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5462000000000004e-05 2024-09-16T03:40:38.442 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.442 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.442 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:38.442 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575671+0000", 2024-09-16T03:40:38.442 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6969999999999998e-06 2024-09-16T03:40:38.442 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.442 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.442 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577693+0000", 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020222439999999999 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577796+0000", 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010306000000000001 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578998+0000", 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012013880000000001 2024-09-16T03:40:38.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.583553+0000", 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045549340000000001 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584072+0000", 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00051922199999999996 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584177+0000", 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105455 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:283 3.4s0 3:2307c0f5:::smithi12138290-28:head [append 0~475136] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.577768+0000", 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.839507205, 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066247500000000004, 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.445 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 283 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577768+0000", 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.446 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577768+0000", 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577770+0000", 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.156e-06 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578319+0000", 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00054978899999999996 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.447 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578336+0000", 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7014e-05 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578346+0000", 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4590000000000004e-06 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580005+0000", 2024-09-16T03:40:38.448 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00165913 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580125+0000", 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000120006 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580986+0000", 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00086071900000000002 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.449 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.583554+0000", 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025686149999999998 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584323+0000", 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076837200000000002 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584393+0000", 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0489999999999998e-05 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.450 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:285 3.4s0 3:2307c0f5:::smithi12138290-28:head [setxattr _header (56) in=7b] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.580152+0000", 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.837123244000001, 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006825065, 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.451 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 285 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580152+0000", 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580152+0000", 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580153+0000", 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580159+0000", 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5280000000000003e-06 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580167+0000", 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3930000000000008e-06 2024-09-16T03:40:38.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580173+0000", 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8969999999999996e-06 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584551+0000", 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043776290000000001 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.454 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.455 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584616+0000", 2024-09-16T03:40:38.455 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5457999999999996e-05 2024-09-16T03:40:38.455 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.455 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.455 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.455 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584936+0000", 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00032023199999999999 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.586559+0000", 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001622559 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.586922+0000", 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00036351599999999998 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.586977+0000", 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5016999999999999e-05 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:286 3.4s0 3:2307c0f5:::smithi12138290-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.580183+0000", 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.837093025, 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079802169999999995, 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.457 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 286 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580183+0000", 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.458 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580183+0000", 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580183+0000", 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4199999999999995e-07 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.459 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580186+0000", 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4430000000000002e-06 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580193+0000", 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0269999999999998e-06 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.580198+0000", 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.079e-06 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.460 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.585225+0000", 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050276269999999998 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.585280+0000", 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4589999999999997e-05 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.587268+0000", 2024-09-16T03:40:38.461 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019877639999999999 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.587356+0000", 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8437999999999996e-05 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.588163+0000", 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080660699999999998 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:311 3.bs0 3:db21b75b:::smithi12138290-3:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.9263327530000005, 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013432349, 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 311 2024-09-16T03:40:38.463 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.464 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490945+0000", 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.419e-06 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490953+0000", 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7379999999999994e-06 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490973+0000", 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0460999999999999e-05 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.465 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490984+0000", 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0857e-05 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491053+0000", 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8351999999999999e-05 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491061+0000", 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3569999999999998e-06 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493588+0000", 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025267430000000001 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493683+0000", 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4876000000000004e-05 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.467 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496393+0000", 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027106500000000002 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504203+0000", 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078100080000000002 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504276+0000", 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2601000000000005e-05 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.468 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504375+0000", 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9287e-05 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:312 3.bs0 3:db21b75b:::smithi12138290-3:head [rollback a] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.9262671279999992, 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013925697000000001, 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.469 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 312 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.470 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491009+0000", 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0200000000000001e-07 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491013+0000", 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8160000000000004e-06 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.471 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.472 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.472 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491022+0000", 2024-09-16T03:40:38.472 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6649999999999992e-06 2024-09-16T03:40:38.472 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.472 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.472 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.472 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491028+0000", 2024-09-16T03:40:38.472 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4760000000000003e-06 2024-09-16T03:40:38.472 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491124+0000", 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6192000000000003e-05 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491132+0000", 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6539999999999999e-06 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.473 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499113+0000", 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079811880000000002 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499184+0000", 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0749999999999999e-05 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499557+0000", 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037325200000000002 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.474 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504380+0000", 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048224590000000003 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504852+0000", 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047194899999999999 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504934+0000", 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2594000000000002e-05 2024-09-16T03:40:38.475 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:313 3.bs0 3:db21b75b:::smithi12138290-3:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.926225487, 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016369406999999999, 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.476 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 313 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.477 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491051+0000", 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.82e-07 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491054+0000", 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5260000000000002e-06 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491063+0000", 2024-09-16T03:40:38.478 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.354e-06 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491069+0000", 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2299999999999996e-06 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491177+0000", 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000108285 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:38.479 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491184+0000", 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0489999999999998e-06 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501958+0000", 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010773448 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502064+0000", 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106661 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.480 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502545+0000", 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000481077 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504381+0000", 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018352970000000001 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.507349+0000", 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002968666 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.481 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.507420+0000", 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0232000000000001e-05 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s4 49/31 ECSubWrite(tid=31, reqid=osd.8.0:31, at_version=49'22, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.495319+0000", 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.9219565670000005, 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061288669999999996, 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.482 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495319+0000", 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495319+0000", 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.483 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495320+0000", 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5700000000000002e-07 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495337+0000", 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7320999999999999e-05 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495348+0000", 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0804e-05 2024-09-16T03:40:38.484 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495353+0000", 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7610000000000004e-06 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495801+0000", 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044771900000000001 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.485 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495811+0000", 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4390000000000006e-06 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501409+0000", 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055981310000000001 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501448+0000", 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9134999999999997e-05 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.486 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:367 3.4s0 3:2ef36fe6:::smithi12138290-31:head [append 0~655360] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.524893+0000", 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8923828629999999, 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081437579999999992, 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.487 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 367 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.524893+0000", 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.524893+0000", 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.524894+0000", 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.609e-06 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.525584+0000", 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00068979999999999996 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.525596+0000", 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1564e-05 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.525603+0000", 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2490000000000004e-06 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.525656+0000", 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2766999999999998e-05 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.525724+0000", 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8411999999999995e-05 2024-09-16T03:40:38.490 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.526523+0000", 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00079852099999999995 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.532730+0000", 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0062072459999999996 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.532950+0000", 2024-09-16T03:40:38.491 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021992600000000001 2024-09-16T03:40:38.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.492 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.492 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533036+0000", 2024-09-16T03:40:38.492 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6663999999999996e-05 2024-09-16T03:40:38.492 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.492 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.492 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:368 3.4s0 3:2ef36fe6:::smithi12138290-31:head [append 0~638976] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.528826+0000", 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8884492939999999, 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057870810000000003, 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 368 2024-09-16T03:40:38.493 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.528826+0000", 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.528826+0000", 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.528828+0000", 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6270000000000001e-06 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529550+0000", 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00072183100000000004 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529561+0000", 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0791999999999999e-05 2024-09-16T03:40:38.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529566+0000", 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2100000000000001e-06 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529598+0000", 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1918000000000002e-05 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529661+0000", 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2966000000000007e-05 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.530356+0000", 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00069555799999999998 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.534159+0000", 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038026190000000001 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.534531+0000", 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037244200000000001 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.534613+0000", 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2118000000000002e-05 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:369 3.4s0 3:2ef36fe6:::smithi12138290-31:head [append 0~131072] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:38.498 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.887695001, 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090412610000000001, 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 369 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2399999999999998e-07 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529661+0000", 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9846999999999996e-05 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529666+0000", 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1919999999999996e-06 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529670+0000", 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3830000000000001e-06 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.530925+0000", 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012551730000000001 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.531047+0000", 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121934 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.531824+0000", 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000777384 2024-09-16T03:40:38.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.534187+0000", 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023626609999999998 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538565+0000", 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043779170000000003 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.503 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538622+0000", 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7246e-05 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:370 3.4s0 3:2ef36fe6:::smithi12138290-31:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.529676+0000", 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8875991970000001, 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092009889999999997, 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 370 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529676+0000", 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529676+0000", 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529677+0000", 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9099999999999999e-07 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529679+0000", 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7799999999999999e-06 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529682+0000", 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4209999999999999e-06 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529685+0000", 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3459999999999998e-06 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533114+0000", 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034284559999999999 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533215+0000", 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010074200000000001 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533660+0000", 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044525100000000002 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.534705+0000", 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001044971 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538781+0000", 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040766639999999998 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538877+0000", 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5966999999999998e-05 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:371 3.4s0 3:2ef36fe6:::smithi12138290-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8875864159999995, 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0095505, 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 371 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:40:38.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:40:38.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7799999999999997e-07 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529691+0000", 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2330000000000001e-06 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529694+0000", 2024-09-16T03:40:38.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5489999999999998e-06 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529697+0000", 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.706e-06 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533902+0000", 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004204564 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533982+0000", 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0399000000000001e-05 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.539228+0000", 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052458579999999999 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.539240+0000", 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1912999999999999e-05 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.515 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:380 3.bs0 3:de0fee80:::smithi12138290-48:head [rollback 3] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.542081+0000", 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8751951010000001, 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006008907, 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.516 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 380 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542081+0000", 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542081+0000", 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542081+0000", 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.58e-07 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542083+0000", 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5830000000000001e-06 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542088+0000", 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4349999999999999e-06 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542091+0000", 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1420000000000002e-06 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.543174+0000", 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001083253 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.543265+0000", 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0866999999999996e-05 2024-09-16T03:40:38.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.543722+0000", 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00045689500000000002 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545478+0000", 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001755768 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.548041+0000", 2024-09-16T03:40:38.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025631870000000002 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.548089+0000", 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8418999999999998e-05 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:381 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.542096+0000", 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8751792759999999, 2024-09-16T03:40:38.521 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086742629999999998, 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 381 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542096+0000", 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542096+0000", 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542097+0000", 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2999999999999999e-07 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542098+0000", 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.22e-06 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542101+0000", 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2590000000000001e-06 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542104+0000", 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7939999999999998e-06 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.524 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544054+0000", 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001950011 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544139+0000", 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4691000000000006e-05 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544560+0000", 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00042157700000000002 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.525 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.547938+0000", 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033775860000000001 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550714+0000", 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027767080000000001 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550771+0000", 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6187000000000003e-05 2024-09-16T03:40:38.526 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:399 3.4s0 3:2307c0f5:::smithi12138290-28:head [read 0~1687552,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.565361+0000", 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8519149759999998, 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0062979280000000004, 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.527 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 399 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565361+0000", 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565361+0000", 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565361+0000", 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7100000000000002e-07 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565364+0000", 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1329999999999999e-06 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565375+0000", 2024-09-16T03:40:38.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1239e-05 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565381+0000", 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3210000000000004e-06 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565703+0000", 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00032207200000000002 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565788+0000", 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4994999999999995e-05 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.571659+0000", 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058706970000000002 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:400 3.4s0 3:2307c0f5:::smithi12138290-28:head [read 0~1687552,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.565388+0000", 2024-09-16T03:40:38.531 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.851887402, 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073422970000000002, 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 400 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565388+0000", 2024-09-16T03:40:38.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565388+0000", 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565389+0000", 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.08e-07 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565391+0000", 2024-09-16T03:40:38.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7489999999999999e-06 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565397+0000", 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0510000000000002e-06 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.565403+0000", 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.13e-06 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.534 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.567732+0000", 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023296049999999998 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.567796+0000", 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4010999999999994e-05 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.572730+0000", 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0049343429999999999 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.535 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.536 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.536 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.536 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:38.538 INFO:tasks.rados.rados.0.smithi121.stdout:174: done (1 left) 2024-09-16T03:40:38.538 INFO:tasks.rados.rados.0.smithi121.stdout:175: done (0 left) 2024-09-16T03:40:38.538 INFO:tasks.rados.rados.0.smithi121.stdout:176: delete oid 7 current snap is 16 2024-09-16T03:40:38.538 INFO:tasks.rados.rados.0.smithi121.stdout:176: done (0 left) 2024-09-16T03:40:38.538 INFO:tasks.rados.rados.0.smithi121.stdout:177: copy_from oid 25 from oid 23 current snap is 16 2024-09-16T03:40:38.538 INFO:tasks.rados.rados.0.smithi121.stdout:178: copy_from oid 26 from oid 29 current snap is 16 2024-09-16T03:40:38.538 INFO:tasks.rados.rados.0.smithi121.stdout:179: snap_remove snap 12 2024-09-16T03:40:38.539 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:38.535+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:38.539 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T03:40:38.539 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:40:38.638 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:38.636+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:38.642 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:38.642 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:38.642 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:38.643 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:38.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:40:38.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:38.739 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:38.738+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:38.794 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:38.794 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:38.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:40:38.804 INFO:teuthology.orchestra.run.smithi103.stderr:Exporting 3.fs1 info 3.fs1( v 53'24 (0'0,53'24] local-lis/les=31/32 n=6 ec=31/31 lis/c=31/31 les/c/f=32/32/0 sis=31) 2024-09-16T03:40:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read 1#3:f06c201b:::smithi12138290-9:head# 2024-09-16T03:40:38.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read 1#3:fb228bb8:::smithi12138290-47:head# 2024-09-16T03:40:38.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read 1#3:fe9db9b8:::smithi12138290-24:4# 2024-09-16T03:40:38.810 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:38.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read 1#3:fe9db9b8:::smithi12138290-24:7# 2024-09-16T03:40:38.810 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:38.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read 1#3:fe9db9b8:::smithi12138290-24:d# 2024-09-16T03:40:38.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read 1#3:fe9db9b8:::smithi12138290-24:head# 2024-09-16T03:40:38.813 INFO:teuthology.orchestra.run.smithi103.stderr:Export successful 2024-09-16T03:40:38.827 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:38.827 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:38.840 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:38.838+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:38.843 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:38.843 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:38.860 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:38.861 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:38.877 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:38.877 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:38.894 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:38.894 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:38.910 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:38.927 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:38.928 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:38.941 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:38.940+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:38.943 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:38.943 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:38.949 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:38.949 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:38.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:38.949 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:38.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.949 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:205 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~655360] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:38.949 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:40:38.949 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 22.423390456, 2024-09-16T03:40:38.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075538640000000004, 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 205 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:40:38.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523025+0000", 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3519999999999999e-06 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523819+0000", 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00079422800000000001 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523834+0000", 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5333e-05 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523842+0000", 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2960000000000001e-06 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523903+0000", 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1124000000000006e-05 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523981+0000", 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8166999999999999e-05 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525035+0000", 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010540969999999999 2024-09-16T03:40:38.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528679+0000", 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036443090000000001 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530521+0000", 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018416789999999999 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530577+0000", 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6279000000000001e-05 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:206 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.525364+0000", 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 22.421049622000002, 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055004260000000001, 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.955 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 206 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525364+0000", 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.956 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525364+0000", 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525365+0000", 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1209999999999999e-06 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525823+0000", 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000457641 2024-09-16T03:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525836+0000", 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2850999999999999e-05 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525842+0000", 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526108+0000", 2024-09-16T03:40:38.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026636800000000001 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.526224+0000", 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116209 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527074+0000", 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00084922299999999997 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528682+0000", 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0016087779999999999 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530821+0000", 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00213875 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530865+0000", 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3625000000000003e-05 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:207 3.6s0 3:656a4357:::smithi12138290-23:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.525856+0000", 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 22.420557712000001, 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054829939999999997, 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 207 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525856+0000", 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525856+0000", 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525857+0000", 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7000000000000004e-07 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525859+0000", 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4380000000000002e-06 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525865+0000", 2024-09-16T03:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6919999999999999e-06 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525870+0000", 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1889999999999998e-06 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527700+0000", 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001829682 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.527793+0000", 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2841000000000001e-05 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528259+0000", 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00046626300000000002 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528747+0000", 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00048799999999999999 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.531285+0000", 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025385569999999999 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.531339+0000", 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3662000000000002e-05 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.966 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:208 3.6s0 3:656a4357:::smithi12138290-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.525875+0000", 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 22.420538371999999, 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006650698, 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 208 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.967 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525875+0000", 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525875+0000", 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525876+0000", 2024-09-16T03:40:38.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525877+0000", 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.477e-06 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525883+0000", 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2689999999999999e-06 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525895+0000", 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2384e-05 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528574+0000", 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002678854 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528669+0000", 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4968000000000003e-05 2024-09-16T03:40:38.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.531658+0000", 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029888029999999999 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.531711+0000", 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3180999999999998e-05 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.532526+0000", 2024-09-16T03:40:38.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00081546099999999996 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(osd.1.22:1 3.6s0 3:616ac6f8:::smithi12138290-8:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.558744+0000", 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 17.387670029999999, 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041018460000000001, 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:38.972 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558744+0000", 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558744+0000", 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.973 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558745+0000", 2024-09-16T03:40:38.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.635e-06 2024-09-16T03:40:38.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558751+0000", 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8610000000000003e-06 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558760+0000", 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6430000000000001e-06 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558768+0000", 2024-09-16T03:40:38.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6490000000000008e-06 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558825+0000", 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7796e-05 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.558951+0000", 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012591100000000001 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.562846+0000", 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038943509999999999 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:305 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:40:38.977 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 10.459164722000001, 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087742589999999995, 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 305 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:40:38.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487251+0000", 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.824e-06 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487259+0000", 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1689999999999992e-06 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487275+0000", 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5889e-05 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487283+0000", 2024-09-16T03:40:38.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9710000000000004e-06 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487385+0000", 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010225099999999999 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487554+0000", 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016910199999999999 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489000+0000", 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001445499 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492361+0000", 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033615799999999999 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495935+0000", 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0035732899999999998 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496023+0000", 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8684000000000007e-05 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:306 3.6s0 3:64535dd7:::smithi12138290-18:head [rollback 5] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 10.45911186, 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0099232220000000006, 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.983 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 306 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487303+0000", 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.44e-07 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487306+0000", 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.817e-06 2024-09-16T03:40:38.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487314+0000", 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4000000000000003e-06 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487319+0000", 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2379999999999997e-06 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489535+0000", 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002216347 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489617+0000", 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1559999999999998e-05 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490002+0000", 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038484000000000001 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492363+0000", 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023609070000000002 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497154+0000", 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047910110000000004 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497225+0000", 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1557999999999995e-05 2024-09-16T03:40:38.988 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:307 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 10.459070044000001, 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010112101, 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.989 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 307 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.51e-07 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487356+0000", 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1793999999999999e-05 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487362+0000", 2024-09-16T03:40:38.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8100000000000003e-06 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487366+0000", 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8670000000000004e-06 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490302+0000", 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029359299999999998 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490396+0000", 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4134000000000004e-05 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490929+0000", 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00053311500000000002 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492364+0000", 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014347279999999999 2024-09-16T03:40:38.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497383+0000", 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050189220000000003 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497456+0000", 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3250000000000005e-05 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:348 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498255+0000", 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.4481585340000001, 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069791749999999998, 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 348 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:38.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498255+0000", 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498255+0000", 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498257+0000", 2024-09-16T03:40:38.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4649999999999999e-06 2024-09-16T03:40:38.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:38.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498264+0000", 2024-09-16T03:40:38.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5440000000000001e-06 2024-09-16T03:40:38.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:38.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498276+0000", 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1324e-05 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498283+0000", 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5730000000000002e-06 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498455+0000", 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000171533 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498560+0000", 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105282 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499260+0000", 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000700042 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505047+0000", 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057870489999999998 2024-09-16T03:40:38.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505134+0000", 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6939000000000002e-05 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505235+0000", 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000100424 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:349 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [rollback 3] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.4481155079999999, 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084181240000000008, 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 349 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.001 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498299+0000", 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6600000000000002e-07 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498301+0000", 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3860000000000001e-06 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498307+0000", 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8590000000000001e-06 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498312+0000", 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6249999999999998e-06 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499617+0000", 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0013049660000000001 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499672+0000", 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5787e-05 2024-09-16T03:40:39.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500118+0000", 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044587499999999998 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505378+0000", 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005259789 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506634+0000", 2024-09-16T03:40:39.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012560169999999999 2024-09-16T03:40:39.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.006 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.006 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506716+0000", 2024-09-16T03:40:39.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2354000000000005e-05 2024-09-16T03:40:39.006 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.006 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.006 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.007 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:350 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:39.007 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:40:39.007 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.448095436, 2024-09-16T03:40:39.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086865829999999995, 2024-09-16T03:40:39.007 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.007 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.007 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:39.007 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:39.007 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 350 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498319+0000", 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.39e-07 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498321+0000", 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.815e-06 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498325+0000", 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8749999999999999e-06 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498329+0000", 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.014e-06 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500398+0000", 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020690069999999999 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500458+0000", 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9429999999999999e-05 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500725+0000", 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026708899999999998 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505380+0000", 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046551040000000002 2024-09-16T03:40:39.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506933+0000", 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015530769999999999 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507005+0000", 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1833000000000002e-05 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s2 53/31 ECSubWrite(tid=40, reqid=osd.13.0:40, at_version=53'20, trim_to=0'0, roll_forward_to=52'18))", 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.4448281039999999, 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007493184, 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.23e-07 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501604+0000", 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8465e-05 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501615+0000", 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.06e-05 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501623+0000", 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8019999999999992e-06 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501779+0000", 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000156093 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501788+0000", 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3840000000000003e-06 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:39.016 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.509033+0000", 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0072446300000000002 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.509079+0000", 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5986999999999999e-05 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.bs6 53/31 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=53'24, trim_to=0'0, roll_forward_to=51'22))", 2024-09-16T03:40:39.017 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.508853+0000", 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.437560854, 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046456709999999997, 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508853+0000", 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508853+0000", 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508853+0000", 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508875+0000", 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2140000000000001e-05 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508884+0000", 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6330000000000002e-06 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508897+0000", 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2807e-05 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508958+0000", 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0810999999999998e-05 2024-09-16T03:40:39.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508966+0000", 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6289999999999999e-06 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513452+0000", 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044858160000000001 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513499+0000", 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6554999999999997e-05 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s2 53/31 ECSubWrite(tid=42, reqid=client.4615.0:362, at_version=53'21, trim_to=0'0, roll_forward_to=53'20))", 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.515864+0000", 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.4305497139999996, 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055959920000000002, 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:39.022 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.515864+0000", 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.515864+0000", 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.515865+0000", 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0900000000000001e-07 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516245+0000", 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038044000000000001 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516260+0000", 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4948e-05 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516269+0000", 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9430000000000006e-06 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516348+0000", 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8817000000000001e-05 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516357+0000", 2024-09-16T03:40:39.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3500000000000003e-06 2024-09-16T03:40:39.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:39.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.521430+0000", 2024-09-16T03:40:39.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050732219999999996 2024-09-16T03:40:39.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.521460+0000", 2024-09-16T03:40:39.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9663e-05 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:412 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~737280] snapc d=[d,b,8,5,3] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.3434826969999998, 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007659028, 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:39.027 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 412 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602932+0000", 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3499999999999995e-07 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603668+0000", 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073570899999999997 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603677+0000", 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4180000000000003e-06 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603702+0000", 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5225999999999999e-05 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603754+0000", 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1391000000000003e-05 2024-09-16T03:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603826+0000", 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2648e-05 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604587+0000", 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076084500000000001 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.608450+0000", 2024-09-16T03:40:39.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038632409999999999 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610532+0000", 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020811580000000001 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610590+0000", 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8656999999999998e-05 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:413 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~573440] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.604921+0000", 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.3414924360000002, 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060231779999999997, 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:39.033 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 413 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604921+0000", 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604921+0000", 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604922+0000", 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0440000000000001e-06 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606010+0000", 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001087743 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606023+0000", 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2384e-05 2024-09-16T03:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606029+0000", 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3160000000000002e-06 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606071+0000", 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1727999999999998e-05 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:39.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606140+0000", 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8895999999999999e-05 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606663+0000", 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00052336799999999997 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610599+0000", 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039365729999999996 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610895+0000", 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00029558699999999999 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610945+0000", 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9539000000000001e-05 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.038 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:414 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~196608] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.606301+0000", 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.3401123789999998, 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055128699999999996, 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 414 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606301+0000", 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606301+0000", 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606302+0000", 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7000000000000004e-07 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.041 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.041 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606513+0000", 2024-09-16T03:40:39.041 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021062399999999999 2024-09-16T03:40:39.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.041 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.041 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.041 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606520+0000", 2024-09-16T03:40:39.041 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0040000000000002e-06 2024-09-16T03:40:39.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606529+0000", 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5750000000000007e-06 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.608634+0000", 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002105015 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.608742+0000", 2024-09-16T03:40:39.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010727500000000001 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.609405+0000", 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00066365700000000001 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610601+0000", 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001195242 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.611743+0000", 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00114207 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.611814+0000", 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1737999999999997e-05 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.044 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:415 3.6s0 3:64535dd7:::smithi12138290-18:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.339876147, 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0056604400000000001, 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 415 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.045 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.53e-07 2024-09-16T03:40:39.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606540+0000", 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0430000000000002e-06 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606545+0000", 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5059999999999997e-06 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606547+0000", 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7530000000000002e-06 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.609850+0000", 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003302471 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.609914+0000", 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3992000000000007e-05 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610241+0000", 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000326767 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610789+0000", 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00054870099999999996 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612138+0000", 2024-09-16T03:40:39.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001348807 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612198+0000", 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0047000000000003e-05 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:416 3.6s0 3:64535dd7:::smithi12138290-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.606551+0000", 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.3398625800000001, 2024-09-16T03:40:39.050 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068823779999999998, 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 416 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606551+0000", 2024-09-16T03:40:39.051 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606551+0000", 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606552+0000", 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5699999999999999e-07 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.052 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606553+0000", 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.297e-06 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606560+0000", 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2919999999999998e-06 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606563+0000", 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0570000000000001e-06 2024-09-16T03:40:39.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.611182+0000", 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046187349999999997 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.611234+0000", 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2080000000000003e-05 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612474+0000", 2024-09-16T03:40:39.054 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001239892 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612554+0000", 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9999000000000004e-05 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.613434+0000", 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00087976899999999997 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.055 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:466 3.6s0 3:6167ac73:::smithi12138290-7:head [stat,delete] snapc 11=[11,10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.47136935000000002, 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0074852260000000002, 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 466 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475046+0000", 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3349999999999999e-06 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475057+0000", 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1158999999999999e-05 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475069+0000", 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1640999999999999e-05 2024-09-16T03:40:39.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475076+0000", 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5379999999999997e-06 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475130+0000", 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4267000000000001e-05 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475135+0000", 2024-09-16T03:40:39.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8640000000000004e-06 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478234+0000", 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030989450000000001 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478358+0000", 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000123666 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:39.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478978+0000", 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00061992000000000004 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.479700+0000", 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00072176900000000003 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.482485+0000", 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027855340000000001 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:39.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:39.062 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:39.062 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.482530+0000", 2024-09-16T03:40:39.062 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4588000000000001e-05 2024-09-16T03:40:39.062 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.062 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.062 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.062 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:39.062 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:39.062 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:39.063 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:39.064 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:39.064 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:39.064 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:39.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:40:39.065 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:39.062+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:39.065 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:39.066 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:39.066 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:39.066 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:39.067 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-09-16T03:40:39.067 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:39.067 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:39.067 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:40:39.068 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:39.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:40:39.164 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:39.162+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:39.184 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:39.184 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:39.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:40:39.265 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:39.264+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:39.296 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:39.296 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:39.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:40:39.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/journal --force --op remove --pgid 3.fs1 2024-09-16T03:40:39.366 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:39.365+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:39.410 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:39.410 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:39.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:40:39.467 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:39.466+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:39.490 INFO:tasks.rados.rados.0.smithi121.stdout:179: done (2 left) 2024-09-16T03:40:39.490 INFO:tasks.rados.rados.0.smithi121.stdout:180: delete oid 40 current snap is 16 2024-09-16T03:40:39.490 INFO:tasks.rados.rados.0.smithi121.stdout:178: finishing copy_from racing read to smithi12138290-26 2024-09-16T03:40:39.490 INFO:tasks.rados.rados.0.smithi121.stdout:178: finishing copy_from to smithi12138290-26 2024-09-16T03:40:39.491 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 26 v 10 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:40:39.491 INFO:tasks.rados.rados.0.smithi121.stdout:177: finishing copy_from racing read to smithi12138290-25 2024-09-16T03:40:39.491 INFO:tasks.rados.rados.0.smithi121.stdout:177: finishing copy_from to smithi12138290-25 2024-09-16T03:40:39.491 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 25 v 16 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-09-16T03:40:39.495 INFO:tasks.rados.rados.0.smithi121.stdout:177: done (2 left) 2024-09-16T03:40:39.495 INFO:tasks.rados.rados.0.smithi121.stdout:178: done (1 left) 2024-09-16T03:40:39.495 INFO:tasks.rados.rados.0.smithi121.stdout:180: done (0 left) 2024-09-16T03:40:39.495 INFO:tasks.rados.rados.0.smithi121.stdout:181: delete oid 13 current snap is 16 2024-09-16T03:40:39.502 INFO:tasks.rados.rados.0.smithi121.stdout:181: done (0 left) 2024-09-16T03:40:39.503 INFO:tasks.rados.rados.0.smithi121.stdout:182: rollback oid 12 current snap is 16 2024-09-16T03:40:39.503 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 12 to 9 2024-09-16T03:40:39.503 INFO:tasks.rados.rados.0.smithi121.stdout:183: rollback oid 44 current snap is 16 2024-09-16T03:40:39.503 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 44 to 9 2024-09-16T03:40:39.503 INFO:tasks.rados.rados.0.smithi121.stdout:184: snap_create 2024-09-16T03:40:39.512 INFO:tasks.rados.rados.0.smithi121.stdout:182: finishing rollback tid 0 to smithi12138290-12 2024-09-16T03:40:39.512 INFO:tasks.rados.rados.0.smithi121.stdout:183: finishing rollback tid 0 to smithi12138290-44 2024-09-16T03:40:39.513 INFO:tasks.rados.rados.0.smithi121.stdout:183: finishing rollback tid 1 to smithi12138290-44 2024-09-16T03:40:39.513 INFO:tasks.rados.rados.0.smithi121.stdout:183: finishing rollback tid 2 to smithi12138290-44 2024-09-16T03:40:39.513 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 44 v 52 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:40:39.513 INFO:tasks.rados.rados.0.smithi121.stdout:182: finishing rollback tid 1 to smithi12138290-12 2024-09-16T03:40:39.513 INFO:tasks.rados.rados.0.smithi121.stdout:182: finishing rollback tid 2 to smithi12138290-12 2024-09-16T03:40:39.513 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 39 (ObjNum 70 snap 5 seq_num 70) dirty exists 2024-09-16T03:40:39.526 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:39.526 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:39.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:40:39.581 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:39.579+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:39.640 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:39.640 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:39.648 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:40:39.670 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:39.668+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:39.770 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:39.768+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:39.792 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:39.792 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:39.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:40:39.871 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:39.870+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:39.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:40:39.973 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:39.971+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:40.055 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:40.055 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:40.064 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:40:40.073 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:40.072+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:40.167 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:40.167 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:40.174 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:40.172+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:40.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:40:40.276 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:40.274+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:40.278 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:40.278 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:40.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:40:40.377 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:40.376+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:40.389 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:40.389 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:40.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:40:40.478 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:40.476+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:40.491 INFO:tasks.rados.rados.0.smithi121.stdout:182: done (2 left) 2024-09-16T03:40:40.492 INFO:tasks.rados.rados.0.smithi121.stdout:183: done (1 left) 2024-09-16T03:40:40.492 INFO:tasks.rados.rados.0.smithi121.stdout:184: done (0 left) 2024-09-16T03:40:40.492 INFO:tasks.rados.rados.0.smithi121.stdout:185: snap_create 2024-09-16T03:40:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:40.510 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:40:40.541 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:40:40.579 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:40.577+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:40.626 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:40.627 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:40.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:40:40.680 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:40.678+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:40.743 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:40.744 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:40.752 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:40:40.796 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:40.795+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:40.856 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:40.857 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:40.866 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:40:40.881 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:40.879+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:40.983 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:40.981+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:41.009 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:41.009 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:41.018 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:40:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: marking collection for removal 2024-09-16T03:40:41.025 INFO:teuthology.orchestra.run.smithi103.stdout:setting '_remove' omap key 2024-09-16T03:40:41.025 INFO:teuthology.orchestra.run.smithi103.stdout:finish_remove_pgs 3.fs1_head removing 3.fs1 2024-09-16T03:40:41.029 INFO:teuthology.orchestra.run.smithi103.stdout:Remove successful 2024-09-16T03:40:41.070 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:41.083 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:41.081+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:41.123 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:41.123 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:41.132 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:40:41.184 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:41.183+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:41.193 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:41.212 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:41.229 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:41.229 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:41.249 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:41.249 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:41.260 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:40:41.261 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:41.261 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:41.278 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:41.278 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:41.285 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:41.283+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:41.295 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:41.295 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:41.312 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:41.312 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:41.329 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:41.329 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:41.346 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:41.346 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:41.363 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:41.366 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:41.366 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:41.375 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:40:41.380 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:41.399 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:41.399 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:41.398+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:41.416 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:41.416 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:41.416 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-09-16T03:40:41.433 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:41.434 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:41.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:41.433+0000 7fc5237fe640 0 -- 172.21.15.76:0/3604090270 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7fc524217c50 msgr2=0x7fc52421d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:41.451 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:41.451 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:40:41.456 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:40:41.469 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T03:40:41.473 INFO:tasks.rados.rados.0.smithi121.stdout:185: done (0 left) 2024-09-16T03:40:41.473 INFO:tasks.rados.rados.0.smithi121.stdout:186: snap_remove snap 13 2024-09-16T03:40:41.487 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:41.484+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:41.490 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:41.490 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:41.490 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:41.490 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:41.490 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:41.499 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:40:41.549 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.fs1.13 2024-09-16T03:40:41.587 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:41.585+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:41.645 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4341.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:39:59.266198+0000", 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 42.376486141000001, 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035726070000000002, 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.646 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4341", 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266198+0000", 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266198+0000", 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266200+0000", 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5620000000000002e-06 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266214+0000", 2024-09-16T03:40:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3709999999999999e-05 2024-09-16T03:40:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266236+0000", 2024-09-16T03:40:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2407000000000001e-05 2024-09-16T03:40:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266247+0000", 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0152999999999999e-05 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266319+0000", 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2133999999999997e-05 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:41.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266328+0000", 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0799999999999995e-06 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.268192+0000", 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018645529999999999 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.268308+0000", 2024-09-16T03:40:41.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011581100000000001 2024-09-16T03:40:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 10", 2024-09-16T03:40:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.268416+0000", 2024-09-16T03:40:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010828300000000001 2024-09-16T03:40:41.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T03:40:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269196+0000", 2024-09-16T03:40:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077989699999999999 2024-09-16T03:40:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269705+0000", 2024-09-16T03:40:41.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00050892999999999999 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269742+0000", 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7128000000000001e-05 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269770+0000", 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7959000000000001e-05 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.654 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4341.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:00.332280+0000", 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.310403721999997, 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034863479999999998, 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4341", 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7 2024-09-16T03:40:41.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332280+0000", 2024-09-16T03:40:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332280+0000", 2024-09-16T03:40:41.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332286+0000", 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9730000000000002e-06 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332296+0000", 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0019e-05 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332314+0000", 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7331000000000001e-05 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332323+0000", 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.414e-06 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332406+0000", 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3013e-05 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.332411+0000", 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5480000000000003e-06 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.333895+0000", 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014844089999999999 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.333953+0000", 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8242999999999998e-05 2024-09-16T03:40:41.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 10", 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.334047+0000", 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3635999999999996e-05 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.334642+0000", 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00059534900000000001 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.335694+0000", 2024-09-16T03:40:41.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001051616 2024-09-16T03:40:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.335728+0000", 2024-09-16T03:40:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3912999999999999e-05 2024-09-16T03:40:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:00.335767+0000", 2024-09-16T03:40:41.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8884000000000002e-05 2024-09-16T03:40:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/31 ECSubWrite(tid=23, reqid=client.4615.0:205, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:41.662 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 25.117493850999999, 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035619599999999999, 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.31e-07 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525645+0000", 2024-09-16T03:40:41.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045454399999999998 2024-09-16T03:40:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525658+0000", 2024-09-16T03:40:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3159999999999999e-05 2024-09-16T03:40:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525669+0000", 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1134999999999999e-05 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525736+0000", 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7175000000000007e-05 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525746+0000", 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.927e-06 2024-09-16T03:40:41.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528720+0000", 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029734119999999999 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528752+0000", 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2475999999999999e-05 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.667 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:229 3.5s0 3:a4bec39a:::smithi12138290-4:head [copy-from ver 9] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 24.102107692000001, 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016562443, 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:41.668 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 229 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540579+0000", 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6910000000000002e-06 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540591+0000", 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2018000000000001e-05 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540631+0000", 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9938999999999998e-05 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540643+0000", 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1902000000000001e-05 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540731+0000", 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8585000000000007e-05 2024-09-16T03:40:41.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540736+0000", 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9919999999999998e-06 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542533+0000", 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017970950000000001 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:41.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542727+0000", 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019304700000000001 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.553519+0000", 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010792568000000001 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554988+0000", 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014687490000000001 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557059+0000", 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00207092 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557139+0000", 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9937000000000004e-05 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:230 3.5s0 3:a4bec39a:::smithi12138290-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 24.101996586999999, 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014084419000000001, 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 230 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540688+0000", 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540692+0000", 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.019e-06 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540724+0000", 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2237000000000003e-05 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540740+0000", 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6075000000000001e-05 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540783+0000", 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2249000000000002e-05 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540786+0000", 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4869999999999998e-06 2024-09-16T03:40:41.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543015+0000", 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022290859999999999 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543063+0000", 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7496000000000001e-05 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554628+0000", 2024-09-16T03:40:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011565203 2024-09-16T03:40:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554697+0000", 2024-09-16T03:40:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9381999999999994e-05 2024-09-16T03:40:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554772+0000", 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4535000000000004e-05 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 42/31 ECSubWrite(tid=30, reqid=osd.14.0:30, at_version=42'20, trim_to=0'0, roll_forward_to=41'18))", 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 20.085109288000002, 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048714179999999998, 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.681 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3799999999999999e-07 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557594+0000", 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9415e-05 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557606+0000", 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1354e-05 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557615+0000", 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9600000000000006e-06 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557669+0000", 2024-09-16T03:40:41.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4024000000000003e-05 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557674+0000", 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.587e-06 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.562413+0000", 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004739254 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.562446+0000", 2024-09-16T03:40:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2586000000000003e-05 2024-09-16T03:40:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:314 3.5s0 3:a4bec39a:::smithi12138290-4:head [stat,delete] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:40:41.686 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.151930595, 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010497464, 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 314 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490755+0000", 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.736e-06 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490767+0000", 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1931e-05 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490783+0000", 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6135000000000001e-05 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490791+0000", 2024-09-16T03:40:41.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1620000000000008e-06 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490849+0000", 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7581000000000003e-05 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490856+0000", 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9299999999999997e-06 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.492657+0000", 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018011539999999999 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.492731+0000", 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4263000000000004e-05 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.493469+0000", 2024-09-16T03:40:41.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073743400000000003 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.495358+0000", 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001888898 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.501195+0000", 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058370879999999998 2024-09-16T03:40:41.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.501251+0000", 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6152e-05 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 53/31 ECSubWrite(tid=51, reqid=client.4615.0:350, at_version=53'39, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.500893+0000", 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1417908959999998, 2024-09-16T03:40:41.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003338826, 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.500893+0000", 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.500893+0000", 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.500894+0000", 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6600000000000002e-07 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.500910+0000", 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6517000000000001e-05 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.500926+0000", 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6370000000000001e-05 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.500934+0000", 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0530000000000007e-06 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.501023+0000", 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8783999999999996e-05 2024-09-16T03:40:41.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.501034+0000", 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0599e-05 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504194+0000", 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031600109999999999 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504232+0000", 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 53/31 ECSubWrite(tid=52, reqid=osd.14.0:52, at_version=53'41, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.502471+0000", 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1402125689999991, 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041823930000000004, 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.698 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502471+0000", 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502471+0000", 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502472+0000", 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.034e-06 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502498+0000", 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5514999999999999e-05 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502510+0000", 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2465e-05 2024-09-16T03:40:41.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502519+0000", 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7460000000000006e-06 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502597+0000", 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8149e-05 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502606+0000", 2024-09-16T03:40:41.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0880000000000001e-06 2024-09-16T03:40:41.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506601+0000", 2024-09-16T03:40:41.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039948939999999997 2024-09-16T03:40:41.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506654+0000", 2024-09-16T03:40:41.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2502e-05 2024-09-16T03:40:41.703 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.703 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.703 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.703 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 53/31 ECSubWrite(tid=53, reqid=osd.14.0:53, at_version=53'43, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T03:40:41.703 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.502542+0000", 2024-09-16T03:40:41.703 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1401420170000005, 2024-09-16T03:40:41.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049945160000000001, 2024-09-16T03:40:41.704 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.704 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.704 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.704 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502542+0000", 2024-09-16T03:40:41.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502542+0000", 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502542+0000", 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502570+0000", 2024-09-16T03:40:41.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7962000000000001e-05 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502578+0000", 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0739999999999993e-06 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502585+0000", 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6089999999999997e-06 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502929+0000", 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034402599999999999 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502938+0000", 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0289999999999994e-06 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507478+0000", 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045398050000000001 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507536+0000", 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8610999999999998e-05 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=27, reqid=client.4615.0:354, at_version=53'8, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1377330659999991, 2024-09-16T03:40:41.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055158589999999997, 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:40:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1e-07 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504976+0000", 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5259999999999999e-05 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504984+0000", 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6850000000000001e-06 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504993+0000", 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4299999999999995e-06 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505310+0000", 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031660299999999999 2024-09-16T03:40:41.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505342+0000", 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.15e-05 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510423+0000", 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050819000000000003 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510467+0000", 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3271e-05 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.505340+0000", 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1373439590000007, 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054441530000000002, 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.713 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.714 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505340+0000", 2024-09-16T03:40:41.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505340+0000", 2024-09-16T03:40:41.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505341+0000", 2024-09-16T03:40:41.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:40:41.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505367+0000", 2024-09-16T03:40:41.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6414000000000002e-05 2024-09-16T03:40:41.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505380+0000", 2024-09-16T03:40:41.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2599000000000001e-05 2024-09-16T03:40:41.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505388+0000", 2024-09-16T03:40:41.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2900000000000002e-06 2024-09-16T03:40:41.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507195+0000", 2024-09-16T03:40:41.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001806698 2024-09-16T03:40:41.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507205+0000", 2024-09-16T03:40:41.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0057999999999999e-05 2024-09-16T03:40:41.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510749+0000", 2024-09-16T03:40:41.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035439759999999999 2024-09-16T03:40:41.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510784+0000", 2024-09-16T03:40:41.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5351000000000001e-05 2024-09-16T03:40:41.722 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.722 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.722 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.722 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=29, reqid=client.4615.0:356, at_version=53'10, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:41.723 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.505935+0000", 2024-09-16T03:40:41.723 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1367484460000004, 2024-09-16T03:40:41.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048734399999999997, 2024-09-16T03:40:41.723 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.723 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.723 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.724 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505935+0000", 2024-09-16T03:40:41.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505935+0000", 2024-09-16T03:40:41.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505936+0000", 2024-09-16T03:40:41.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1999999999999999e-07 2024-09-16T03:40:41.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505955+0000", 2024-09-16T03:40:41.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8726999999999999e-05 2024-09-16T03:40:41.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505970+0000", 2024-09-16T03:40:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5254e-05 2024-09-16T03:40:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505981+0000", 2024-09-16T03:40:41.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0421000000000001e-05 2024-09-16T03:40:41.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507941+0000", 2024-09-16T03:40:41.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019600279999999999 2024-09-16T03:40:41.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507952+0000", 2024-09-16T03:40:41.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1524e-05 2024-09-16T03:40:41.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510788+0000", 2024-09-16T03:40:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002835522 2024-09-16T03:40:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510809+0000", 2024-09-16T03:40:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1243999999999999e-05 2024-09-16T03:40:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=30, reqid=client.4615.0:357, at_version=53'12, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.508937+0000", 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1337465550000001, 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037675410000000001, 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508937+0000", 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508937+0000", 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508938+0000", 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1999999999999998e-07 2024-09-16T03:40:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509070+0000", 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013158500000000001 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509081+0000", 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1303e-05 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509089+0000", 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7510000000000008e-06 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509138+0000", 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9014000000000002e-05 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509146+0000", 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2330000000000006e-06 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512665+0000", 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00351914 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512705+0000", 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9595000000000003e-05 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=31, reqid=client.4615.0:358, at_version=53'13, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.511311+0000", 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1313726590000002, 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004370306, 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511311+0000", 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511311+0000", 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511312+0000", 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8800000000000002e-07 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511418+0000", 2024-09-16T03:40:41.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010646599999999999 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511430+0000", 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1222e-05 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511439+0000", 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9679999999999995e-06 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511584+0000", 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000145315 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511596+0000", 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2329e-05 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515644+0000", 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00404762 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515682+0000", 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7697999999999997e-05 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=32, reqid=client.4615.0:359, at_version=53'14, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.513931+0000", 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1287532700000007, 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004942264, 2024-09-16T03:40:41.741 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513931+0000", 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513931+0000", 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513932+0000", 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514010+0000", 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8285e-05 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514023+0000", 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2666999999999999e-05 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514032+0000", 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3999999999999998e-06 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514382+0000", 2024-09-16T03:40:41.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000349742 2024-09-16T03:40:41.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514391+0000", 2024-09-16T03:40:41.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.986e-06 2024-09-16T03:40:41.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518837+0000", 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044466339999999997 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518873+0000", 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5605999999999998e-05 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.746 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s1 53/31 ECSubWrite(tid=19, reqid=client.4615.0:375, at_version=53'12, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:41.747 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.539243+0000", 2024-09-16T03:40:41.747 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1034409279999995, 2024-09-16T03:40:41.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043355520000000003, 2024-09-16T03:40:41.747 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.747 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.747 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.747 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539243+0000", 2024-09-16T03:40:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539243+0000", 2024-09-16T03:40:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539248+0000", 2024-09-16T03:40:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8420000000000004e-06 2024-09-16T03:40:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539870+0000", 2024-09-16T03:40:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062189099999999996 2024-09-16T03:40:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539884+0000", 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4644000000000001e-05 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539894+0000", 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2390000000000008e-06 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539963+0000", 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9496e-05 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539974+0000", 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0592e-05 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543541+0000", 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035675329999999999 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543579+0000", 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7314999999999998e-05 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 54/31 ECSubWrite(tid=37, reqid=osd.12.0:37, at_version=54'19, trim_to=0'0, roll_forward_to=53'15))", 2024-09-16T03:40:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:33.511068+0000", 2024-09-16T03:40:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1316156569999993, 2024-09-16T03:40:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003243021, 2024-09-16T03:40:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511068+0000", 2024-09-16T03:40:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511068+0000", 2024-09-16T03:40:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511069+0000", 2024-09-16T03:40:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5499999999999999e-07 2024-09-16T03:40:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511090+0000", 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1708999999999998e-05 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511099+0000", 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4430000000000003e-06 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511105+0000", 2024-09-16T03:40:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.703e-06 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511675+0000", 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056975199999999998 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511686+0000", 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0373999999999999e-05 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.514274+0000", 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002588861 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.514311+0000", 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6823999999999998e-05 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:467 3.5s0 3:a45e0081:::smithi12138290-25:head [copy-from ver 16] snapc 11=[11,10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.1598528510000001, 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014721097000000001, 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 467 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482832+0000", 2024-09-16T03:40:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3e-06 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482839+0000", 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3010000000000003e-06 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482849+0000", 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9499999999999996e-06 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482854+0000", 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1410000000000004e-06 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482916+0000", 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2040999999999998e-05 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483028+0000", 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112674 2024-09-16T03:40:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.491088+0000", 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080595969999999999 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.493079+0000", 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019907990000000001 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.497458+0000", 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004378804 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.497552+0000", 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4489999999999998e-05 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:468 3.5s0 3:a45e0081:::smithi12138290-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:40:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.1598008719999999, 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010159837999999999, 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 468 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482884+0000", 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.16e-07 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482888+0000", 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0099999999999997e-06 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482903+0000", 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5418999999999999e-05 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482910+0000", 2024-09-16T03:40:41.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6850000000000004e-06 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483315+0000", 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040511699999999998 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483369+0000", 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3884000000000001e-05 2024-09-16T03:40:41.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492657+0000", 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092879590000000001 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492897+0000", 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023993300000000001 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.493043+0000", 2024-09-16T03:40:41.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014601499999999999 2024-09-16T03:40:41.770 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.770 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.770 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.770 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:41.770 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:41.770 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:41.771 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:41.768+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:41.772 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:40:41.872 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:41.871+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:41.875 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:41.876 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:41.884 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:40:41.974 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:41.972+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:41.989 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:41.989 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:41.989 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:41.990 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:41.990 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:41.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:40:42.075 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:42.073+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:42.103 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:253 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 20.521168359000001, 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084206999999999997, 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.104 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 253 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579061+0000", 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3350000000000001e-06 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579072+0000", 2024-09-16T03:40:42.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0964999999999999e-05 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579093+0000", 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1766000000000001e-05 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579104+0000", 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0271999999999999e-05 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579200+0000", 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5917999999999999e-05 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579208+0000", 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9479999999999997e-06 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581270+0000", 2024-09-16T03:40:42.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020611140000000002 2024-09-16T03:40:42.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581437+0000", 2024-09-16T03:40:42.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016734000000000001 2024-09-16T03:40:42.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587479+0000", 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060420420000000001 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:254 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 20.521085273000001, 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087869469999999998, 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.110 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 254 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579142+0000", 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579146+0000", 2024-09-16T03:40:42.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0600000000000001e-06 2024-09-16T03:40:42.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579155+0000", 2024-09-16T03:40:42.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7620000000000001e-06 2024-09-16T03:40:42.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579161+0000", 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8640000000000001e-06 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579289+0000", 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000128677 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579298+0000", 2024-09-16T03:40:42.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4100000000000008e-06 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581916+0000", 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026177570000000001 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581995+0000", 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9833999999999995e-05 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587928+0000", 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059329329999999996 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:255 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 20.521041356000001, 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090005769999999992, 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.116 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 255 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579186+0000", 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9500000000000002e-07 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579189+0000", 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T03:40:42.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579197+0000", 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8189999999999992e-06 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579202+0000", 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579337+0000", 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134547 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579343+0000", 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583029+0000", 2024-09-16T03:40:42.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036861540000000001 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583113+0000", 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3370999999999998e-05 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.588186+0000", 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050730269999999999 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:354 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.498416+0000", 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.6018108780000002, 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017750095, 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.122 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 354 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498416+0000", 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498416+0000", 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498418+0000", 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0899999999999999e-06 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498426+0000", 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.909e-06 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498442+0000", 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6164999999999999e-05 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498451+0000", 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5240000000000006e-06 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498523+0000", 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1482000000000004e-05 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498535+0000", 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2072000000000001e-05 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502229+0000", 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003693806 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502333+0000", 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104041 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503454+0000", 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011210440000000001 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515617+0000", 2024-09-16T03:40:42.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012162681999999999 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516067+0000", 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000450817 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516166+0000", 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8462999999999996e-05 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.128 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.129 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:355 3.ds0 3:b952e66c:::smithi12138290-29:head [rollback b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:42.129 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:40:42.129 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.6017520059999999, 2024-09-16T03:40:42.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017895208999999999, 2024-09-16T03:40:42.129 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.129 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.129 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 355 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8100000000000003e-07 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498478+0000", 2024-09-16T03:40:42.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8049999999999998e-06 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498485+0000", 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0219999999999999e-06 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498489+0000", 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9829999999999998e-06 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498604+0000", 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011544699999999999 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498609+0000", 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2760000000000002e-06 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503989+0000", 2024-09-16T03:40:42.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053800360000000004 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504091+0000", 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010255300000000001 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504699+0000", 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00060748699999999998 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515811+0000", 2024-09-16T03:40:42.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011112662000000001 2024-09-16T03:40:42.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:42.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516302+0000", 2024-09-16T03:40:42.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049017500000000003 2024-09-16T03:40:42.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516370+0000", 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8281999999999994e-05 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:356 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.498505+0000", 2024-09-16T03:40:42.136 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.6017213570000006, 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018089424, 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 356 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498505+0000", 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498505+0000", 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498506+0000", 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3700000000000001e-07 2024-09-16T03:40:42.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498508+0000", 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498513+0000", 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4519999999999996e-06 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498518+0000", 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6399999999999996e-06 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498643+0000", 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125265 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498648+0000", 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0320000000000003e-06 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505217+0000", 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065689839999999999 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505317+0000", 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9613999999999999e-05 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505895+0000", 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057821100000000002 2024-09-16T03:40:42.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515868+0000", 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099727659999999992 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516509+0000", 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00064155600000000005 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516595+0000", 2024-09-16T03:40:42.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5206999999999999e-05 2024-09-16T03:40:42.143 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.143 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.143 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.143 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:357 3.ds0 3:b84aed6c:::smithi12138290-13:head [append 0~622592] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:42.143 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.504582+0000", 2024-09-16T03:40:42.143 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5956447849999993, 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012468976999999999, 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 357 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504582+0000", 2024-09-16T03:40:42.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504582+0000", 2024-09-16T03:40:42.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504584+0000", 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.057e-06 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505874+0000", 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012901060000000001 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505888+0000", 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3689999999999999e-05 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505896+0000", 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3580000000000003e-06 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506261+0000", 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000365239 2024-09-16T03:40:42.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506357+0000", 2024-09-16T03:40:42.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5466999999999999e-05 2024-09-16T03:40:42.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508262+0000", 2024-09-16T03:40:42.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001905622 2024-09-16T03:40:42.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516183+0000", 2024-09-16T03:40:42.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079204419999999998 2024-09-16T03:40:42.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:42.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516969+0000", 2024-09-16T03:40:42.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00078583600000000004 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517051+0000", 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2159999999999999e-05 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.150 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 53/31 ECSubWriteReply(tid=27, last_complete=53'8, committed=1, applied=1))", 2024-09-16T03:40:42.151 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.507210+0000", 2024-09-16T03:40:42.151 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5930169880000005, 2024-09-16T03:40:42.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027545360000000001, 2024-09-16T03:40:42.151 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.151 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.151 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.152 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507210+0000", 2024-09-16T03:40:42.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507210+0000", 2024-09-16T03:40:42.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507211+0000", 2024-09-16T03:40:42.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5200000000000001e-06 2024-09-16T03:40:42.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507218+0000", 2024-09-16T03:40:42.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.109e-06 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507233+0000", 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4579999999999999e-05 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507243+0000", 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8949999999999993e-06 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509929+0000", 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026860959999999998 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509964+0000", 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5336000000000002e-05 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:358 3.ds0 3:b84aed6c:::smithi12138290-13:head [append 0~475136] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.507540+0000", 2024-09-16T03:40:42.155 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5926869979999996, 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011581542, 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 358 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507540+0000", 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507540+0000", 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507541+0000", 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.716e-06 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507949+0000", 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040747899999999998 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507963+0000", 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3828999999999999e-05 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507971+0000", 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7430000000000008e-06 2024-09-16T03:40:42.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510078+0000", 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002106293 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510200+0000", 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012225100000000001 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511236+0000", 2024-09-16T03:40:42.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010360289999999999 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516185+0000", 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049487079999999996 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519053+0000", 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028678610000000002 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519121+0000", 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8633000000000005e-05 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:359 3.ds0 3:b84aed6c:::smithi12138290-13:head [append 0~294912] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.509631+0000", 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5905951849999997, 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097547659999999998, 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.161 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 359 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509631+0000", 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509631+0000", 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509634+0000", 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1459999999999999e-06 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509870+0000", 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023675700000000001 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509882+0000", 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1905999999999999e-05 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509891+0000", 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9870000000000005e-06 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512302+0000", 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002410812 2024-09-16T03:40:42.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512443+0000", 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014075700000000001 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513286+0000", 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000843021 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516670+0000", 2024-09-16T03:40:42.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033843889999999998 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519344+0000", 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026737129999999999 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519386+0000", 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2277999999999997e-05 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.166 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:360 3.ds0 3:b84aed6c:::smithi12138290-13:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5903179680000008, 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010089805, 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 360 2024-09-16T03:40:42.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8999999999999996e-07 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509913+0000", 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0269999999999997e-06 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509922+0000", 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.687e-06 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509928+0000", 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8000000000000004e-06 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517293+0000", 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073646559999999998 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517402+0000", 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109687 2024-09-16T03:40:42.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517900+0000", 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049748700000000002 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519217+0000", 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001317033 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:42.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519941+0000", 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00072411000000000003 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519998+0000", 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7627999999999998e-05 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:361 3.ds0 3:b84aed6c:::smithi12138290-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.509950+0000", 2024-09-16T03:40:42.172 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5902761509999994, 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01277462, 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 361 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509950+0000", 2024-09-16T03:40:42.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509950+0000", 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509951+0000", 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7299999999999996e-07 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509956+0000", 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4739999999999999e-06 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509963+0000", 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509968+0000", 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4530000000000001e-06 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518702+0000", 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087336250000000001 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518806+0000", 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104905 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.520329+0000", 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015226879999999999 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.520422+0000", 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3224000000000001e-05 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.522725+0000", 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023026980000000002 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.177 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 53/31 ECSubWriteReply(tid=30, last_complete=53'12, committed=1, applied=1))", 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.512606+0000", 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5876203320000002, 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030753960000000002, 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512606+0000", 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512606+0000", 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512607+0000", 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2600000000000002e-07 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512612+0000", 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.305e-06 2024-09-16T03:40:42.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512624+0000", 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2053e-05 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512634+0000", 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515651+0000", 2024-09-16T03:40:42.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030175610000000002 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515682+0000", 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0501000000000002e-05 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 53/31 ECSubWriteReply(tid=30, last_complete=53'12, committed=1, applied=1))", 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.512762+0000", 2024-09-16T03:40:42.181 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5874646200000004, 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029636369999999999, 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512762+0000", 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512762+0000", 2024-09-16T03:40:42.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512763+0000", 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7899999999999997e-07 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512768+0000", 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3689999999999998e-06 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512780+0000", 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1698e-05 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512788+0000", 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7579999999999992e-06 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515708+0000", 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029204270000000002 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515726+0000", 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7606000000000001e-05 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 53/31 ECSubWriteReply(tid=30, last_complete=53'12, committed=1, applied=1))", 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.512839+0000", 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5873874229999991, 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029290829999999999, 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.185 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512839+0000", 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512839+0000", 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512840+0000", 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5700000000000001e-07 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512857+0000", 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6827999999999999e-05 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512867+0000", 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417999999999999e-05 2024-09-16T03:40:42.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512874+0000", 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8340000000000001e-06 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515751+0000", 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028768980000000001 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515768+0000", 2024-09-16T03:40:42.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7248000000000002e-05 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 53/31 ECSubWriteReply(tid=29, last_complete=53'10, committed=1, applied=1))", 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.513740+0000", 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5864865730000002, 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028676790000000001, 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513740+0000", 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513740+0000", 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513740+0000", 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.29e-07 2024-09-16T03:40:42.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513742+0000", 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1210000000000001e-06 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513751+0000", 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5920000000000001e-06 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513758+0000", 2024-09-16T03:40:42.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516430+0000", 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026726089999999998 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516608+0000", 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000177388 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.192 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 53/31 ECSubWriteReply(tid=31, last_complete=53'13, committed=1, applied=1))", 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.513917+0000", 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5863093740000007, 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027555140000000001, 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513917+0000", 2024-09-16T03:40:42.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513917+0000", 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513918+0000", 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6600000000000002e-07 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513923+0000", 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513938+0000", 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4669e-05 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513954+0000", 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5597999999999999e-05 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516649+0000", 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026954869999999999 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516673+0000", 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3683999999999999e-05 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s6 53/31 ECSubWrite(tid=19, reqid=client.4615.0:375, at_version=53'12, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.539609+0000", 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5606175709999999, 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039602220000000002, 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539609+0000", 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539609+0000", 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539609+0000", 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.49e-07 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540054+0000", 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000444988 2024-09-16T03:40:42.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540068+0000", 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3388e-05 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540077+0000", 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1349999999999998e-06 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540156+0000", 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9769000000000001e-05 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540166+0000", 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8090000000000004e-06 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543538+0000", 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033718910000000001 2024-09-16T03:40:42.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543569+0000", 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1093000000000002e-05 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s6 53/31 ECSubWrite(tid=22, reqid=client.4615.0:377, at_version=0'0, trim_to=0'0, roll_forward_to=53'14))", 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.551278+0000", 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5489490769999996, 2024-09-16T03:40:42.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034132730000000001, 2024-09-16T03:40:42.202 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.202 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.202 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.202 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551278+0000", 2024-09-16T03:40:42.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551278+0000", 2024-09-16T03:40:42.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551279+0000", 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5290000000000001e-06 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551293+0000", 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.415e-05 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551316+0000", 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2552e-05 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551327+0000", 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0858e-05 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551400+0000", 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3107000000000003e-05 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551409+0000", 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3840000000000003e-06 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554640+0000", 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032307239999999999 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554691+0000", 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0968999999999999e-05 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.207 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.cs2 53/31 ECSubWrite(tid=92, reqid=client.4615.0:388, at_version=53'27, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.552782+0000", 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5474445879999994, 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037395129999999999, 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552782+0000", 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552782+0000", 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552783+0000", 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1000000000000001e-06 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552866+0000", 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2631000000000002e-05 2024-09-16T03:40:42.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552887+0000", 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1715000000000001e-05 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552898+0000", 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.08e-05 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552979+0000", 2024-09-16T03:40:42.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1166000000000002e-05 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552990+0000", 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0679000000000001e-05 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.556491+0000", 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035008600000000002 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:42.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:42.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.556522+0000", 2024-09-16T03:40:42.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0562000000000003e-05 2024-09-16T03:40:42.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.212 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:42.212 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:42.212 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:42.214 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:40:42.214 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:42.212+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:42.314 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:42.312+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:42.324 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:42.324 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:42.324 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:42.324 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:42.333 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:40:42.415 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:42.414+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:42.436 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:42.436 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:42.445 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:40:42.474 INFO:tasks.rados.rados.0.smithi121.stdout:186: done (0 left) 2024-09-16T03:40:42.474 INFO:tasks.rados.rados.0.smithi121.stdout:187: copy_from oid 33 from oid 48 current snap is 18 2024-09-16T03:40:42.475 INFO:tasks.rados.rados.0.smithi121.stdout:188: snap_create 2024-09-16T03:40:42.495 INFO:tasks.rados.rados.0.smithi121.stdout:187: finishing copy_from racing read to smithi12138290-33 2024-09-16T03:40:42.495 INFO:tasks.rados.rados.0.smithi121.stdout:187: finishing copy_from to smithi12138290-33 2024-09-16T03:40:42.496 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 33 v 28 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:40:42.516 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:42.514+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:42.555 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:42.555 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:42.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs5 38/31 ECSubWrite(tid=63, reqid=client.4615.0:226, at_version=38'10, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.544170+0000", 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.008515068000001, 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056030519999999999, 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544170+0000", 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544170+0000", 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544171+0000", 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4300000000000003e-07 2024-09-16T03:40:42.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544310+0000", 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013949599999999999 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544330+0000", 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9539e-05 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544342+0000", 2024-09-16T03:40:42.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.155e-05 2024-09-16T03:40:42.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544395+0000", 2024-09-16T03:40:42.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3318e-05 2024-09-16T03:40:42.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544404+0000", 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.149e-06 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549741+0000", 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005337456 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549773+0000", 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1900999999999998e-05 2024-09-16T03:40:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.561 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.561 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.561 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:251 3.0s0 3:00cc7e13:::smithi12138290-42:head [copy-from ver 4] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:42.561 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:40:42.561 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 20.995949568, 2024-09-16T03:40:42.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013120588000000001, 2024-09-16T03:40:42.561 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.561 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 251 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556738+0000", 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8989999999999999e-06 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556745+0000", 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:40:42.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556759+0000", 2024-09-16T03:40:42.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.451e-05 2024-09-16T03:40:42.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556766+0000", 2024-09-16T03:40:42.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1260000000000003e-06 2024-09-16T03:40:42.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556817+0000", 2024-09-16T03:40:42.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1693999999999997e-05 2024-09-16T03:40:42.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:42.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556824+0000", 2024-09-16T03:40:42.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8549999999999996e-06 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557940+0000", 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001115888 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558009+0000", 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8707999999999993e-05 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564426+0000", 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064170659999999999 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.566682+0000", 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022565039999999999 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569779+0000", 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030962770000000001 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569856+0000", 2024-09-16T03:40:42.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7843999999999995e-05 2024-09-16T03:40:42.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.568 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.568 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:252 3.0s0 3:00cc7e13:::smithi12138290-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:42.568 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.556781+0000", 2024-09-16T03:40:42.569 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 20.995904437, 2024-09-16T03:40:42.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008452701, 2024-09-16T03:40:42.569 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.569 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.569 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:42.569 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.569 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.569 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.570 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 252 2024-09-16T03:40:42.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.570 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556781+0000", 2024-09-16T03:40:42.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556781+0000", 2024-09-16T03:40:42.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556782+0000", 2024-09-16T03:40:42.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8899999999999999e-07 2024-09-16T03:40:42.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556786+0000", 2024-09-16T03:40:42.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0910000000000003e-06 2024-09-16T03:40:42.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556795+0000", 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1249999999999999e-06 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556801+0000", 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1040000000000003e-06 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556876+0000", 2024-09-16T03:40:42.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5113999999999999e-05 2024-09-16T03:40:42.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:42.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556880+0000", 2024-09-16T03:40:42.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2749999999999997e-06 2024-09-16T03:40:42.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558172+0000", 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001291936 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558261+0000", 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8628999999999995e-05 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.565110+0000", 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068490360000000002 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.565168+0000", 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8087999999999997e-05 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.565234+0000", 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5714000000000001e-05 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.577 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:332 3.0s0 3:0cbe5f28:::smithi12138290-39:head [append 0~589824] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:42.577 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.475060+0000", 2024-09-16T03:40:42.577 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.077625657, 2024-09-16T03:40:42.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0090762959999999993, 2024-09-16T03:40:42.577 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.577 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.577 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 332 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475060+0000", 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475060+0000", 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475062+0000", 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.644e-06 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475787+0000", 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00072438999999999999 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475800+0000", 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3288000000000001e-05 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475807+0000", 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1779999999999997e-06 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475858+0000", 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.062e-05 2024-09-16T03:40:42.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475864+0000", 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3589999999999996e-06 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.477601+0000", 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001736455 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.477664+0000", 2024-09-16T03:40:42.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2928999999999993e-05 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478538+0000", 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00087489000000000002 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.480310+0000", 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001771044 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:42.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.484058+0000", 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037481099999999998 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.484136+0000", 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8388999999999998e-05 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.583 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:333 3.0s0 3:0cbe5f28:::smithi12138290-39:head [append 0~573440] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.478105+0000", 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.074580183, 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006433935, 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 333 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478105+0000", 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478105+0000", 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478106+0000", 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1200000000000001e-06 2024-09-16T03:40:42.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478666+0000", 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00055984100000000005 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478679+0000", 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3067999999999999e-05 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478688+0000", 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0180000000000007e-06 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479162+0000", 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000474177 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479259+0000", 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6261999999999994e-05 2024-09-16T03:40:42.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479878+0000", 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061973200000000001 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.482152+0000", 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022740899999999999 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.484493+0000", 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023406490000000002 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.484539+0000", 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5977999999999999e-05 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:334 3.0s0 3:0cbe5f28:::smithi12138290-39:head [append 0~196608] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.479263+0000", 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.07342188, 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093778520000000008, 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 334 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.590 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479263+0000", 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479263+0000", 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479264+0000", 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9999999999999995e-07 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479450+0000", 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018548100000000001 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479460+0000", 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8030000000000008e-06 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479465+0000", 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9049999999999996e-06 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.480432+0000", 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00096703299999999996 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.480512+0000", 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9793e-05 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.480999+0000", 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048716200000000002 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.488494+0000", 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074957039999999997 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.488566+0000", 2024-09-16T03:40:42.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1384000000000006e-05 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.488641+0000", 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5587000000000007e-05 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:335 3.0s0 3:0cbe5f28:::smithi12138290-39:head [setxattr _header (57) in=7b] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.479477+0000", 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.073208491000001, 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096011870000000006, 2024-09-16T03:40:42.595 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 335 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479477+0000", 2024-09-16T03:40:42.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479477+0000", 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479477+0000", 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7100000000000002e-07 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479480+0000", 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.711e-06 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479486+0000", 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5879999999999997e-06 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479489+0000", 2024-09-16T03:40:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2399999999999999e-06 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481425+0000", 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019356620000000001 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481515+0000", 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0655000000000005e-05 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481869+0000", 2024-09-16T03:40:42.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00035413799999999999 2024-09-16T03:40:42.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.488659+0000", 2024-09-16T03:40:42.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067899839999999998 2024-09-16T03:40:42.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.488716+0000", 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6205999999999997e-05 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.489078+0000", 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036243199999999999 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.602 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:336 3.0s0 3:0cbe5f28:::smithi12138290-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:42.602 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.479494+0000", 2024-09-16T03:40:42.602 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.073191156, 2024-09-16T03:40:42.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01140001, 2024-09-16T03:40:42.602 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.602 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.602 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 336 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479494+0000", 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479494+0000", 2024-09-16T03:40:42.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479495+0000", 2024-09-16T03:40:42.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4900000000000001e-07 2024-09-16T03:40:42.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479497+0000", 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1260000000000001e-06 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479503+0000", 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479506+0000", 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.482263+0000", 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027574219999999998 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.482312+0000", 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9196999999999997e-05 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.489119+0000", 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068063610000000004 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.489197+0000", 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8570000000000002e-05 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.490894+0000", 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001697034 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.607 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 53/31 ECSubWrite(tid=50, reqid=client.4615.0:349, at_version=53'38, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.500322+0000", 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.052363664, 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0061250009999999997, 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500322+0000", 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500322+0000", 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500323+0000", 2024-09-16T03:40:42.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.153e-06 2024-09-16T03:40:42.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500345+0000", 2024-09-16T03:40:42.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2157000000000001e-05 2024-09-16T03:40:42.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500358+0000", 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2986999999999999e-05 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500367+0000", 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1500000000000005e-06 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.501187+0000", 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081999099999999995 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.501198+0000", 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.046e-05 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.506404+0000", 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052068180000000002 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.506447+0000", 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2284999999999999e-05 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 53/31 ECSubWrite(tid=51, reqid=client.4615.0:350, at_version=53'39, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T03:40:42.613 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.500839+0000", 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.051846456, 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056341550000000001, 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500839+0000", 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500839+0000", 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500839+0000", 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8299999999999997e-07 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500859+0000", 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9170000000000001e-05 2024-09-16T03:40:42.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500873+0000", 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3893000000000001e-05 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500881+0000", 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4659999999999999e-06 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.501722+0000", 2024-09-16T03:40:42.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00084063499999999995 2024-09-16T03:40:42.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.501731+0000", 2024-09-16T03:40:42.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6080000000000002e-06 2024-09-16T03:40:42.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.506451+0000", 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047198630000000004 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.506473+0000", 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1937e-05 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.3s4 53/31 ECSubWrite(tid=19, reqid=client.4615.0:375, at_version=53'12, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.539176+0000", 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.013509679, 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052376710000000002, 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539176+0000", 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539176+0000", 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539176+0000", 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9499999999999998e-07 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539681+0000", 2024-09-16T03:40:42.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050504399999999996 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539693+0000", 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1770000000000001e-05 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539701+0000", 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5609999999999999e-06 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539762+0000", 2024-09-16T03:40:42.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0242999999999999e-05 2024-09-16T03:40:42.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539772+0000", 2024-09-16T03:40:42.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0129000000000001e-05 2024-09-16T03:40:42.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.544371+0000", 2024-09-16T03:40:42.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045991950000000004 2024-09-16T03:40:42.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.544413+0000", 2024-09-16T03:40:42.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2333999999999998e-05 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:421 3.0s0 3:0cbe5f28:::smithi12138290-39:head [stat,delete] snapc d=[b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.512028+0000", 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.0406573310000002, 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065887159999999997, 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.624 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 421 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512028+0000", 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512028+0000", 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512030+0000", 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1349999999999999e-06 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512036+0000", 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8839999999999999e-06 2024-09-16T03:40:42.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512051+0000", 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4698999999999999e-05 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512058+0000", 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2540000000000003e-06 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512113+0000", 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5098e-05 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512118+0000", 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.305e-06 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.513450+0000", 2024-09-16T03:40:42.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001331558 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.513661+0000", 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000210869 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514391+0000", 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073004199999999997 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518031+0000", 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003639626 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518526+0000", 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00049509499999999998 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518617+0000", 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1150999999999995e-05 2024-09-16T03:40:42.630 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.bs4 54/31 ECSubWrite(tid=66, reqid=osd.10.0:66, at_version=54'30, trim_to=0'0, roll_forward_to=53'28))", 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.514028+0000", 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.0386570880000008, 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00691765, 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514028+0000", 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514028+0000", 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514029+0000", 2024-09-16T03:40:42.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8100000000000002e-07 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514047+0000", 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8420999999999999e-05 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514056+0000", 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7660000000000004e-06 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514063+0000", 2024-09-16T03:40:42.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3829999999999997e-06 2024-09-16T03:40:42.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514106+0000", 2024-09-16T03:40:42.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3544999999999997e-05 2024-09-16T03:40:42.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514113+0000", 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5050000000000004e-06 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520899+0000", 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067864600000000002 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520946+0000", 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6888999999999998e-05 2024-09-16T03:40:42.635 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.636 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.636 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.636 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 54/31 ECSubWrite(tid=62, reqid=osd.14.0:62, at_version=54'52, trim_to=0'0, roll_forward_to=53'48))", 2024-09-16T03:40:42.636 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.514225+0000", 2024-09-16T03:40:42.636 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.0384602889999996, 2024-09-16T03:40:42.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067025690000000002, 2024-09-16T03:40:42.636 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.637 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.637 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.637 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514225+0000", 2024-09-16T03:40:42.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514225+0000", 2024-09-16T03:40:42.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514226+0000", 2024-09-16T03:40:42.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.82e-07 2024-09-16T03:40:42.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514242+0000", 2024-09-16T03:40:42.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6246e-05 2024-09-16T03:40:42.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514250+0000", 2024-09-16T03:40:42.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9659999999999996e-06 2024-09-16T03:40:42.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514256+0000", 2024-09-16T03:40:42.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9569999999999999e-06 2024-09-16T03:40:42.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514309+0000", 2024-09-16T03:40:42.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3075999999999998e-05 2024-09-16T03:40:42.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514321+0000", 2024-09-16T03:40:42.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2337999999999999e-05 2024-09-16T03:40:42.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520884+0000", 2024-09-16T03:40:42.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065623139999999996 2024-09-16T03:40:42.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520928+0000", 2024-09-16T03:40:42.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.409e-05 2024-09-16T03:40:42.643 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.643 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.643 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.644 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.3s4 54/31 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=54'16, trim_to=0'0, roll_forward_to=53'14))", 2024-09-16T03:40:42.644 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.515287+0000", 2024-09-16T03:40:42.644 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.0373979030000005, 2024-09-16T03:40:42.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056953539999999997, 2024-09-16T03:40:42.644 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.644 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.644 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.644 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515287+0000", 2024-09-16T03:40:42.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515287+0000", 2024-09-16T03:40:42.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515288+0000", 2024-09-16T03:40:42.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6199999999999999e-07 2024-09-16T03:40:42.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515311+0000", 2024-09-16T03:40:42.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3591000000000002e-05 2024-09-16T03:40:42.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515324+0000", 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2583e-05 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515335+0000", 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0883999999999999e-05 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515402+0000", 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7241000000000002e-05 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515412+0000", 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0482999999999999e-05 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520951+0000", 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055382469999999996 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520983+0000", 2024-09-16T03:40:42.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2162999999999997e-05 2024-09-16T03:40:42.650 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.650 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.650 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.651 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s5 54/31 ECSubWrite(tid=57, reqid=osd.6.0:56, at_version=0'0, trim_to=0'0, roll_forward_to=54'45))", 2024-09-16T03:40:42.651 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.518434+0000", 2024-09-16T03:40:42.651 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.0342511850000005, 2024-09-16T03:40:42.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055100909999999999, 2024-09-16T03:40:42.652 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.652 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.652 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:42.652 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518434+0000", 2024-09-16T03:40:42.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518434+0000", 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518435+0000", 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3099999999999998e-07 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518445+0000", 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.043e-05 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518460+0000", 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5260999999999998e-05 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518468+0000", 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2309999999999999e-06 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518527+0000", 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9704999999999998e-05 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518536+0000", 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3189999999999995e-06 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.523821+0000", 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052849619999999998 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.523944+0000", 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123752 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:460 3.0s0 3:08509fe0:::smithi12138290-16:head [append 0~458752] snapc 10=[10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:37.475645+0000", 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.07703989, 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059849059999999999, 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 460 2024-09-16T03:40:42.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.475645+0000", 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.475645+0000", 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.475647+0000", 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.53e-06 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.476110+0000", 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00046279300000000003 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.476126+0000", 2024-09-16T03:40:42.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6399999999999999e-05 2024-09-16T03:40:42.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.476134+0000", 2024-09-16T03:40:42.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5809999999999999e-06 2024-09-16T03:40:42.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.476187+0000", 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3325999999999997e-05 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.476193+0000", 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3570000000000003e-06 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.477725+0000", 2024-09-16T03:40:42.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001532013 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.477793+0000", 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7631000000000004e-05 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.478606+0000", 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081251800000000003 2024-09-16T03:40:42.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479831+0000", 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012256700000000001 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.481537+0000", 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001706068 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.481630+0000", 2024-09-16T03:40:42.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3018999999999998e-05 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:461 3.0s0 3:08509fe0:::smithi12138290-16:head [append 0~688128] snapc 10=[10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:37.479182+0000", 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.0735037729999997, 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053366230000000004, 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.664 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 461 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479182+0000", 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479182+0000", 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479183+0000", 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2470000000000001e-06 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479896+0000", 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071288299999999998 2024-09-16T03:40:42.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479908+0000", 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2478e-05 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479916+0000", 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9589999999999995e-06 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479966+0000", 2024-09-16T03:40:42.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9975999999999997e-05 2024-09-16T03:40:42.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.480029+0000", 2024-09-16T03:40:42.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3003999999999995e-05 2024-09-16T03:40:42.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.480645+0000", 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061600999999999995 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.481642+0000", 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00099715099999999994 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484466+0000", 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028235180000000001 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484518+0000", 2024-09-16T03:40:42.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2397e-05 2024-09-16T03:40:42.670 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.670 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.670 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.670 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:463 3.0s0 3:08509fe0:::smithi12138290-16:head [setxattr _header (57) in=7b] snapc 10=[10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:37.482677+0000", 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.0700078980000001, 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00709101, 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 463 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482677+0000", 2024-09-16T03:40:42.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482677+0000", 2024-09-16T03:40:42.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482678+0000", 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4199999999999995e-07 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482681+0000", 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0290000000000001e-06 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482686+0000", 2024-09-16T03:40:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.164e-06 2024-09-16T03:40:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482698+0000", 2024-09-16T03:40:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1635e-05 2024-09-16T03:40:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.483801+0000", 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001103436 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.483877+0000", 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5174999999999998e-05 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484163+0000", 2024-09-16T03:40:42.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00028641499999999999 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.486805+0000", 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026422709999999999 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.489716+0000", 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029110749999999999 2024-09-16T03:40:42.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.489768+0000", 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2167999999999999e-05 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:464 3.0s0 3:08509fe0:::smithi12138290-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:37.482705+0000", 2024-09-16T03:40:42.677 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.0699800049999997, 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0081540769999999992, 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 464 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482705+0000", 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482705+0000", 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482706+0000", 2024-09-16T03:40:42.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2800000000000002e-07 2024-09-16T03:40:42.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:42.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482713+0000", 2024-09-16T03:40:42.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.759e-06 2024-09-16T03:40:42.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:42.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482719+0000", 2024-09-16T03:40:42.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-09-16T03:40:42.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:42.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482722+0000", 2024-09-16T03:40:42.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1219999999999999e-06 2024-09-16T03:40:42.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484586+0000", 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001864386 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484638+0000", 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1718000000000002e-05 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.490065+0000", 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054270029999999997 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.490116+0000", 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0519999999999997e-05 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.490859+0000", 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00074374800000000002 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:42.683 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:42.684 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:42.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:40:42.686 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:42.683+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:42.786 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:42.784+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:42.787 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:42.788 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:42.788 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:42.788 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:42.796 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:40:42.887 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:42.885+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:42.900 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:42.900 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:42.900 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:42.900 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:42.900 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:42.908 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:40:42.988 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:42.985+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:43.015 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:43.015 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:43.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:43.016 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:43.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.016 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:39 1.0 e25/24 1:30306672:devicehealth::main.db.0000000000000000:head v 25'25, mlcod=25'22)", 2024-09-16T03:40:43.017 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:43.017 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 44.694427038000001, 2024-09-16T03:40:43.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042816670000000003, 2024-09-16T03:40:43.017 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.017 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.018 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.018 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:43.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317956+0000", 2024-09-16T03:40:43.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.317957+0000", 2024-09-16T03:40:43.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.54e-07 2024-09-16T03:40:43.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318024+0000", 2024-09-16T03:40:43.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6730000000000007e-05 2024-09-16T03:40:43.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318033+0000", 2024-09-16T03:40:43.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1589999999999999e-06 2024-09-16T03:40:43.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318038+0000", 2024-09-16T03:40:43.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8040000000000001e-06 2024-09-16T03:40:43.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318080+0000", 2024-09-16T03:40:43.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2475000000000002e-05 2024-09-16T03:40:43.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.318093+0000", 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3315e-05 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.322203+0000", 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041094620000000004 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:58.322238+0000", 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4968000000000001e-05 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:164 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'118, mlcod=25'113)", 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.854149903, 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047753090000000002, 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.026 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158233+0000", 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158234+0000", 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8100000000000003e-07 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158294+0000", 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0359999999999998e-05 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158301+0000", 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3159999999999999e-06 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158305+0000", 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3900000000000002e-06 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158343+0000", 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7793000000000002e-05 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.158361+0000", 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8230999999999999e-05 2024-09-16T03:40:43.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.162979+0000", 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046178230000000001 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.163009+0000", 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9915000000000001e-05 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:178 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'128, mlcod=25'125)", 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.728745039000003, 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058842529999999999, 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283638+0000", 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283639+0000", 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283643+0000", 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3980000000000001e-06 2024-09-16T03:40:43.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283652+0000", 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6430000000000001e-06 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283656+0000", 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7339999999999999e-06 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283698+0000", 2024-09-16T03:40:43.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1785000000000001e-05 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.283715+0000", 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6555e-05 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289498+0000", 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0057832869999999998 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289522+0000", 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4290999999999998e-05 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:179 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'129, mlcod=25'125)", 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.728273909999999, 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066372109999999996, 2024-09-16T03:40:43.035 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284110+0000", 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.89e-07 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284175+0000", 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5667000000000007e-05 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284186+0000", 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.009e-05 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284190+0000", 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7600000000000002e-06 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284222+0000", 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1888999999999999e-05 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284234+0000", 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.225e-05 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290722+0000", 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064871909999999998 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290746+0000", 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4875000000000002e-05 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.039 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:180 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'130, mlcod=25'125)", 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.727659600999999, 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072760109999999998, 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.68e-07 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284729+0000", 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4009999999999999e-06 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284738+0000", 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6940000000000007e-06 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284743+0000", 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5549999999999996e-06 2024-09-16T03:40:43.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286547+0000", 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0018044770000000001 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286567+0000", 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9272000000000001e-05 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.291958+0000", 2024-09-16T03:40:43.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053911540000000004 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292000+0000", 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.189e-05 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:181 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'131, mlcod=25'126)", 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:43.044 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.727119051000003, 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067648710000000004, 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:40:43.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285265+0000", 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6100000000000001e-07 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285269+0000", 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5020000000000003e-06 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285279+0000", 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6879999999999994e-06 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285286+0000", 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4590000000000001e-06 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286842+0000", 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015560000000000001 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286854+0000", 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1959e-05 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292001+0000", 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051468859999999998 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292029+0000", 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7815999999999999e-05 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.048 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:182 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'132, mlcod=25'126)", 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.726674330999998, 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0063484680000000003, 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2200000000000004e-07 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285737+0000", 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7886999999999999e-05 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285747+0000", 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9459999999999993e-06 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285752+0000", 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:40:43.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287024+0000", 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012715409999999999 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287042+0000", 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8919999999999998e-05 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292031+0000", 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049888179999999999 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292057+0000", 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6057000000000001e-05 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:183 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'133, mlcod=25'126)", 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:43.053 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.725979236999997, 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060351210000000001, 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286404+0000", 2024-09-16T03:40:43.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286405+0000", 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4300000000000001e-06 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286429+0000", 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3459999999999999e-05 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286439+0000", 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0105999999999999e-05 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286443+0000", 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.527e-06 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289328+0000", 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028841940000000001 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289340+0000", 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1861e-05 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292411+0000", 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030717399999999999 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292439+0000", 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7803000000000001e-05 2024-09-16T03:40:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:184 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'134, mlcod=25'126)", 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.725564869999999, 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056342509999999998, 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286818+0000", 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286819+0000", 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5499999999999998e-07 2024-09-16T03:40:43.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286823+0000", 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2889999999999999e-06 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286833+0000", 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4769999999999992e-06 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286839+0000", 2024-09-16T03:40:43.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6599999999999998e-06 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289572+0000", 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002733161 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289593+0000", 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0494999999999999e-05 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292440+0000", 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028470660000000001 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292453+0000", 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2548000000000001e-05 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:185 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'135, mlcod=25'126)", 2024-09-16T03:40:43.062 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.725001118000002, 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050824039999999996, 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287382+0000", 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287383+0000", 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287410+0000", 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6985e-05 2024-09-16T03:40:43.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287419+0000", 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6879999999999994e-06 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287424+0000", 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6829999999999999e-06 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289884+0000", 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002459639 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.289917+0000", 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3303999999999998e-05 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292453+0000", 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025360769999999999 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292464+0000", 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1364e-05 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:186 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'136, mlcod=25'126)", 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.724486486000004, 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045805179999999996, 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.067 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287897+0000", 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3200000000000005e-07 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287902+0000", 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.634e-06 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287912+0000", 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7739999999999999e-06 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287916+0000", 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6099999999999999e-06 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290274+0000", 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023574519999999999 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290289+0000", 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5481e-05 2024-09-16T03:40:43.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292466+0000", 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021765439999999999 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292477+0000", 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:204 1.0 e26/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'150, mlcod=25'145)", 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.601118315999997, 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051027959999999997, 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411265+0000", 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411272+0000", 2024-09-16T03:40:43.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1029999999999998e-06 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411280+0000", 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3129999999999999e-06 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411287+0000", 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2570000000000001e-06 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411347+0000", 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0214000000000003e-05 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411361+0000", 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3735e-05 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416329+0000", 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049679889999999999 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416368+0000", 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8634999999999999e-05 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:205 1.0 e26/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'151, mlcod=26'147)", 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.600579048, 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004582226, 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.076 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411804+0000", 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411805+0000", 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2699999999999999e-07 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411808+0000", 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6710000000000001e-06 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411815+0000", 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9480000000000002e-06 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411820+0000", 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0270000000000003e-06 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411851+0000", 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0474e-05 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.411862+0000", 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.171e-05 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416370+0000", 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045072810000000001 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:00.416386+0000", 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6588000000000001e-05 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es2 37/31 ECSubWrite(tid=26, reqid=client.4615.0:218, at_version=37'8, trim_to=0'0, roll_forward_to=34'5))", 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:16.556575+0000", 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.455807891999999, 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044039439999999999, 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556575+0000", 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556575+0000", 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556578+0000", 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8490000000000002e-06 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556597+0000", 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8513999999999999e-05 2024-09-16T03:40:43.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556607+0000", 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0309000000000001e-05 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556616+0000", 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3889999999999994e-06 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556677+0000", 2024-09-16T03:40:43.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1066000000000001e-05 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.556686+0000", 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.952e-06 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560938+0000", 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042516860000000002 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:16.560979+0000", 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1179e-05 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.5s6 42/31 ECSubWrite(tid=23, reqid=osd.8.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.555040+0000", 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.457343467000001, 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005476687, 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.085 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555040+0000", 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555040+0000", 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555041+0000", 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0500000000000002e-07 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555051+0000", 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0305e-05 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555063+0000", 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2002e-05 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555069+0000", 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5889999999999999e-06 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555132+0000", 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2514000000000005e-05 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555142+0000", 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0118999999999999e-05 2024-09-16T03:40:43.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560482+0000", 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053403239999999996 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560516+0000", 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3929e-05 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 42/31 ECSubWrite(tid=73, reqid=osd.0.0:73, at_version=42'13, trim_to=0'0, roll_forward_to=38'11))", 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.456464119, 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045524069999999996, 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555919+0000", 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7700000000000001e-07 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555946+0000", 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6560999999999999e-05 2024-09-16T03:40:43.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555959+0000", 2024-09-16T03:40:43.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3042e-05 2024-09-16T03:40:43.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.555968+0000", 2024-09-16T03:40:43.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1079999999999999e-06 2024-09-16T03:40:43.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556053+0000", 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4975999999999994e-05 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556063+0000", 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6120000000000005e-06 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560429+0000", 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043663369999999996 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.560471+0000", 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2493999999999997e-05 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 42/31 ECSubWrite(tid=74, reqid=osd.0.0:73, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:43.094 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.449508733999998, 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064234579999999999, 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562875+0000", 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562885+0000", 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0069e-05 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562897+0000", 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1619e-05 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562903+0000", 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.652e-06 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562965+0000", 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1576999999999997e-05 2024-09-16T03:40:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562975+0000", 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0314e-05 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569190+0000", 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062148569999999998 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569298+0000", 2024-09-16T03:40:43.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010764600000000001 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/31 ECSubWrite(tid=89, reqid=osd.0.0:88, at_version=0'0, trim_to=0'0, roll_forward_to=53'24))", 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.500490+0000", 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.511893509, 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048251589999999999, 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.099 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500490+0000", 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500490+0000", 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500491+0000", 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9999999999999995e-07 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500506+0000", 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5305e-05 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500519+0000", 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.349e-05 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500531+0000", 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1719000000000001e-05 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500605+0000", 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3694000000000003e-05 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500615+0000", 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9080000000000007e-06 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505262+0000", 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046472570000000001 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505315+0000", 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2785999999999999e-05 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.103 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.4s6 53/31 ECSubWrite(tid=56, reqid=client.4615.0:370, at_version=53'16, trim_to=0'0, roll_forward_to=53'13))", 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.533706+0000", 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.478677209000001, 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046483239999999997, 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533706+0000", 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533706+0000", 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533707+0000", 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7499999999999999e-07 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533723+0000", 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6269e-05 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533748+0000", 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4736000000000002e-05 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533757+0000", 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2739999999999996e-06 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533826+0000", 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9439999999999999e-05 2024-09-16T03:40:43.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.533839+0000", 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2962999999999999e-05 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.538317+0000", 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044773130000000001 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.538354+0000", 2024-09-16T03:40:43.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7554e-05 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.4s6 53/31 ECSubWrite(tid=57, reqid=client.4615.0:370, at_version=0'0, trim_to=0'0, roll_forward_to=53'16))", 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.539071+0000", 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.473311748, 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045011890000000001, 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.108 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539071+0000", 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539071+0000", 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539072+0000", 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3099999999999996e-07 2024-09-16T03:40:43.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539082+0000", 2024-09-16T03:40:43.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7869999999999996e-06 2024-09-16T03:40:43.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539097+0000", 2024-09-16T03:40:43.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4498000000000001e-05 2024-09-16T03:40:43.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539104+0000", 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9619999999999993e-06 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539165+0000", 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0754999999999997e-05 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539175+0000", 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5449999999999992e-06 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.543533+0000", 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043584690000000002 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.543573+0000", 2024-09-16T03:40:43.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9341999999999997e-05 2024-09-16T03:40:43.113 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.113 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.113 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.113 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.113 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.113 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:43.114 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:40:43.115 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:43.113+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:43.215 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:43.213+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:43.218 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:43.219 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:43.219 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:43.219 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:43.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:40:43.258 INFO:teuthology.orchestra.run.smithi103.stdout:get_pg_num_history pg_num_history pg_num_history(e53 pg_nums {1={24=1},2={24=8},3={31=16}} deleted_pools ) 2024-09-16T03:40:43.258 INFO:teuthology.orchestra.run.smithi103.stdout:Importing pgid 3.fs1 2024-09-16T03:40:43.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write 1#3:f06c201b:::smithi12138290-9:head# 2024-09-16T03:40:43.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-09-16T03:40:43.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write 1#3:fb228bb8:::smithi12138290-47:head# 2024-09-16T03:40:43.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-09-16T03:40:43.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write 1#3:fe9db9b8:::smithi12138290-24:4# 2024-09-16T03:40:43.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write 1#3:fe9db9b8:::smithi12138290-24:7# 2024-09-16T03:40:43.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write 1#3:fe9db9b8:::smithi12138290-24:d# 2024-09-16T03:40:43.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write 1#3:fe9db9b8:::smithi12138290-24:head# 2024-09-16T03:40:43.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset d=[]:{4=[3],7=[5],d=[d,b,8]} 2024-09-16T03:40:43.262 INFO:teuthology.orchestra.run.smithi103.stdout:write_pg epoch 53 info 3.fs1( v 53'24 (0'0,53'24] local-lis/les=31/32 n=6 ec=31/31 lis/c=31/31 les/c/f=32/32/0 sis=31) 2024-09-16T03:40:43.263 INFO:teuthology.orchestra.run.smithi103.stdout:Import successful 2024-09-16T03:40:43.316 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:43.314+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:43.333 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:43.333 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:43.334 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:43.334 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:43.334 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:43.342 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:40:43.426 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:43.424+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:43.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:43.477 INFO:tasks.rados.rados.0.smithi121.stdout:187: done (1 left) 2024-09-16T03:40:43.477 INFO:tasks.rados.rados.0.smithi121.stdout:188: done (0 left) 2024-09-16T03:40:43.477 INFO:tasks.rados.rados.0.smithi121.stdout:189: snap_create 2024-09-16T03:40:43.484 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:231 3.7s0 3:e441e0cd:::smithi12138290-34:head [append 0~737280] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.548164+0000", 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.938222601, 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074938289999999996, 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.489 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 231 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.548164+0000", 2024-09-16T03:40:43.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.548164+0000", 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.548166+0000", 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4929999999999999e-06 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549023+0000", 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00085696500000000001 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549041+0000", 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8612000000000001e-05 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549051+0000", 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5389999999999996e-06 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549108+0000", 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6780999999999997e-05 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549115+0000", 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3690000000000001e-06 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550938+0000", 2024-09-16T03:40:43.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001823376 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.551048+0000", 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010907499999999999 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.552071+0000", 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0010230149999999999 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555488+0000", 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034173379999999998 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555573+0000", 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4932000000000006e-05 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555658+0000", 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5334000000000007e-05 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.495 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:232 3.7s0 3:e441e0cd:::smithi12138290-34:head [append 0~376832] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.550228+0000", 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.936158915, 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058243390000000004, 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:43.496 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 232 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550228+0000", 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550228+0000", 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550229+0000", 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2999999999999999e-07 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550657+0000", 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042805700000000003 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550670+0000", 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2796000000000001e-05 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550678+0000", 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9219999999999997e-06 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.553144+0000", 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024667650000000001 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.553239+0000", 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4949000000000002e-05 2024-09-16T03:40:43.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.553968+0000", 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00072873999999999997 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555765+0000", 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001796874 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556000+0000", 2024-09-16T03:40:43.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000235443 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556052+0000", 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1863000000000001e-05 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:233 3.7s0 3:e441e0cd:::smithi12138290-34:head [setxattr _header (56) in=7b] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.550693+0000", 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.935694308999999, 2024-09-16T03:40:43.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062768190000000003, 2024-09-16T03:40:43.503 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.503 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.503 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.503 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:43.503 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:43.503 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 233 2024-09-16T03:40:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550693+0000", 2024-09-16T03:40:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550693+0000", 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550693+0000", 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6700000000000003e-07 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550697+0000", 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4170000000000001e-06 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550703+0000", 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6830000000000002e-06 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550709+0000", 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1660000000000002e-06 2024-09-16T03:40:43.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.554595+0000", 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038863970000000002 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.554682+0000", 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7299999999999994e-05 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555082+0000", 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00039952299999999999 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555766+0000", 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00068426399999999997 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556891+0000", 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001124508 2024-09-16T03:40:43.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556969+0000", 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8893999999999994e-05 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:234 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.550720+0000", 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.935666892, 2024-09-16T03:40:43.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075177949999999999, 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 234 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550720+0000", 2024-09-16T03:40:43.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550720+0000", 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550721+0000", 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9200000000000001e-07 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550723+0000", 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2409999999999998e-06 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550730+0000", 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8750000000000002e-06 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550736+0000", 2024-09-16T03:40:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8579999999999996e-06 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556094+0000", 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053587419999999997 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556171+0000", 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7232999999999997e-05 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.557335+0000", 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011640000000000001 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.557405+0000", 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9979000000000002e-05 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.558238+0000", 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000832375 2024-09-16T03:40:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.0s1 42/31 ECSubWrite(tid=20, reqid=client.4615.0:251, at_version=42'11, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.564475+0000", 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.921911676000001, 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050564989999999999, 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564475+0000", 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564475+0000", 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564476+0000", 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7300000000000002e-07 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564523+0000", 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7376999999999997e-05 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564537+0000", 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4503e-05 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564556+0000", 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8417e-05 2024-09-16T03:40:43.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564627+0000", 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1507000000000005e-05 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564640+0000", 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2629000000000001e-05 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569498+0000", 2024-09-16T03:40:43.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048575579999999997 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569532+0000", 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4235e-05 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:264 3.7s0 3:e06e137e:::smithi12138290-27:head [copy-from ver 4] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.895710498, 2024-09-16T03:40:43.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012313443, 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 264 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:40:43.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590679+0000", 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1739999999999999e-06 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590686+0000", 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.43e-06 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590709+0000", 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2748999999999999e-05 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590718+0000", 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0359999999999995e-06 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590860+0000", 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000142208 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590966+0000", 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000106439 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.599409+0000", 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008442583 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600331+0000", 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000922202 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.602907+0000", 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002575787 2024-09-16T03:40:43.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.602990+0000", 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2835000000000002e-05 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:265 3.7s0 3:e06e137e:::smithi12138290-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.895588659000001, 2024-09-16T03:40:43.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0092800720000000003, 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 265 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590799+0000", 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590804+0000", 2024-09-16T03:40:43.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3009999999999997e-06 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590832+0000", 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7851000000000002e-05 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590849+0000", 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6691e-05 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.591224+0000", 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037502100000000001 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.591256+0000", 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.184e-05 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.599957+0000", 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087011659999999998 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600004+0000", 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7218000000000001e-05 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600078+0000", 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4065000000000003e-05 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:296 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:40:43.530 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.015268340999999, 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084845249999999997, 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 296 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471120+0000", 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.804e-06 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471128+0000", 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0199999999999994e-06 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471144+0000", 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5957999999999999e-05 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471153+0000", 2024-09-16T03:40:43.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7299999999999994e-06 2024-09-16T03:40:43.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471238+0000", 2024-09-16T03:40:43.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4860999999999999e-05 2024-09-16T03:40:43.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:43.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471244+0000", 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0859999999999998e-06 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.473493+0000", 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022493750000000001 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.473624+0000", 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130891 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.479603+0000", 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059788000000000003 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:297 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:40:43.536 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.01521194, 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094313820000000003, 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 297 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9900000000000001e-07 2024-09-16T03:40:43.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471180+0000", 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4619999999999998e-06 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471186+0000", 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3130000000000004e-06 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471190+0000", 2024-09-16T03:40:43.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.194e-06 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471314+0000", 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012333299999999999 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471322+0000", 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8280000000000003e-06 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.476742+0000", 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054202740000000001 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.476849+0000", 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010663899999999999 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480606+0000", 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037578400000000001 2024-09-16T03:40:43.541 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:298 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.015163413, 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011872347, 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 298 2024-09-16T03:40:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:40:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2399999999999999e-07 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471227+0000", 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1149999999999998e-06 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471232+0000", 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.152e-06 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471237+0000", 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9540000000000003e-06 2024-09-16T03:40:43.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471356+0000", 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011896900000000001 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471364+0000", 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480706+0000", 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093421570000000002 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480820+0000", 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011411100000000001 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.483096+0000", 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022753819999999998 2024-09-16T03:40:43.547 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:308 3.7s0 3:e441e0cd:::smithi12138290-34:head [create,setxattr h^blbcaNtSmmsQtZPdaRtPkhOOgN_Vi\\f`NtsZ[V (561) in=40b,setxattr T (17332) in=1b,setxattr UVoaZteYfmpthpcfl[gebaR\\TdqcNkhPRRdrgsX] (279) in=40b,setxattr bTPgfXb\\YXtXkQag\\Qh`hlXr[posd]]cje]^eebl (1121) in=40b,setxattr hm^gakWSOOOfNOTph^tma]iaNsaqfjqZYb\\seZNW (3297) in=40b,setxattr YiNn[]m_q`YlQlaV^mRn^ (17332) in=21b,setxattr \\tllpkXThmWXSkOopogrdtjY]f^a^sem_krafjfi (947) in=40b,setxattr RXiRZRQVSaUTaZmQT^bfqZhS^mZsqadaPdW]ijm` (1963) in=40b,setxattr WTmsTZhNpU`tZY\\^aROaNdmtYsjjsgWWboehPOP` (19508) in=40b,setxattr UZmWZeWaoiX]jmbn][UadnNkcZc_hdcjidtNsRYo (1999) in=40b,setxattr NSebjRYYQ`SbaatVjZWNab`cenkljRq`^]Xr\\jRm (11788) in=40b,setxattr ip`daUaop[rdbNPo\\^c[Wcelsl]a][RScQZ[kfa^ (18524) in=40b,setxattr qU_qcNcQpNkegs_Od`jieTXX[mto]lsR^jTcYprR (593) in=40b,setxattr dboVg`srq[bReagdR]fX\\Pe`Warrp]\\[XpSipTN[ (1683) in=40b,setxattr ZdTaVj[ScjsOiQ]a_]R[RQWoYohnjd`fi[dYmfs` (840) in=40b,setxattr N_[blmbpstWhtUgW]ZjZ[\\O_erqcnQN]R_V^U\\RT (1015) in=40b,setxattr jXbP_mNsadnq`UVfmc^lfZUs\\r_[ZnrahqTUk^Zp (383) in=40b,setxattr nmemd\\ipehj\\q^UVY_YtSrhZjcabNXhlai_noQSO (807) in=40b,setxattr Rbc\\cahhlcrckUioieUbQnrseNkZkdPf\\rUmNb]S (384) in=40b,setxattr ZiOkXqg^PobTdZgrtZ]]l^Rap_tfSQoUQqaefj]] (69) in=40b] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 14.995902485, 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013488215, 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 308 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490486+0000", 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.601e-06 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490741+0000", 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00025543300000000001 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490756+0000", 2024-09-16T03:40:43.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4949e-05 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490764+0000", 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1659999999999997e-06 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.492684+0000", 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019205509999999999 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:43.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.493193+0000", 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050880499999999998 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.494675+0000", 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001482443 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.496347+0000", 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001671358 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.503822+0000", 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074755639999999996 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.503973+0000", 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000150345 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.0s1 52/31 ECSubWrite(tid=25, reqid=client.4615.0:334, at_version=52'14, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T03:40:43.553 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.481008+0000", 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.005378689, 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053169339999999997, 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481008+0000", 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481008+0000", 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481009+0000", 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9100000000000003e-07 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481064+0000", 2024-09-16T03:40:43.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4620000000000002e-05 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481075+0000", 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1053000000000001e-05 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481082+0000", 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4560000000000003e-06 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481131+0000", 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9168000000000001e-05 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481137+0000", 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8760000000000002e-06 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.486292+0000", 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051551729999999999 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.486325+0000", 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2897e-05 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.0s1 52/31 ECSubWrite(tid=26, reqid=client.4615.0:335, at_version=52'15, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.481900+0000", 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.004486442999999, 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044494519999999996, 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.558 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481900+0000", 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481900+0000", 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481901+0000", 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6199999999999997e-07 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481917+0000", 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5811000000000002e-05 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481930+0000", 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3497e-05 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481939+0000", 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0940000000000008e-06 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481989+0000", 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0003999999999998e-05 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481994+0000", 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8359999999999997e-06 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.486329+0000", 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043346069999999999 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.486350+0000", 2024-09-16T03:40:43.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0941000000000002e-05 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.ds2 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.505108+0000", 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.981278855999999, 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046943510000000003, 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.563 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505108+0000", 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505108+0000", 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505109+0000", 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0240000000000001e-06 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505132+0000", 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2670000000000001e-05 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505151+0000", 2024-09-16T03:40:43.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8975e-05 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505158+0000", 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3000000000000004e-06 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505216+0000", 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8352e-05 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.505240+0000", 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3754e-05 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.509755+0000", 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045143680000000004 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.509802+0000", 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7908000000000003e-05 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.9s3 53/31 ECSubWrite(tid=44, reqid=client.4615.0:364, at_version=53'23, trim_to=0'0, roll_forward_to=53'20))", 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.520717+0000", 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.965669985, 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050875099999999999, 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.568 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520717+0000", 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520717+0000", 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520727+0000", 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6039999999999999e-06 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520779+0000", 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2864000000000001e-05 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520793+0000", 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3585e-05 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520804+0000", 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1498000000000001e-05 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520897+0000", 2024-09-16T03:40:43.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2527000000000004e-05 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.520913+0000", 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6222000000000002e-05 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.525776+0000", 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048629440000000001 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.525804+0000", 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8266e-05 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s4 54/31 ECSubWrite(tid=56, reqid=osd.6.0:56, at_version=54'45, trim_to=0'0, roll_forward_to=53'43))", 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.510581+0000", 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.9758060959999995, 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051499390000000001, 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.573 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510581+0000", 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510581+0000", 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510582+0000", 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9600000000000008e-07 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510622+0000", 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0048e-05 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510642+0000", 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9984999999999999e-05 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510652+0000", 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0068000000000001e-05 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510846+0000", 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000193573 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510857+0000", 2024-09-16T03:40:43.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1163e-05 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515690+0000", 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048328490000000002 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515731+0000", 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1257000000000002e-05 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.577 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.ds2 54/31 ECSubWrite(tid=37, reqid=osd.12.0:37, at_version=54'19, trim_to=0'0, roll_forward_to=53'15))", 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.510873+0000", 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.9755139600000007, 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048256439999999996, 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510873+0000", 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510873+0000", 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510874+0000", 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7000000000000003e-07 2024-09-16T03:40:43.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510897+0000", 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3433000000000001e-05 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510912+0000", 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4245e-05 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.510921+0000", 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.166e-06 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.511029+0000", 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010799299999999999 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.511038+0000", 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1079999999999999e-06 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515663+0000", 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046254470000000004 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515699+0000", 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5382000000000001e-05 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s4 54/31 ECSubWrite(tid=57, reqid=osd.6.0:56, at_version=0'0, trim_to=0'0, roll_forward_to=54'45))", 2024-09-16T03:40:43.582 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.518458+0000", 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.9679287260000002, 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005602709, 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518458+0000", 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518458+0000", 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518459+0000", 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518471+0000", 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1647999999999999e-05 2024-09-16T03:40:43.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518484+0000", 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3621000000000001e-05 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518494+0000", 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7399999999999999e-06 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518574+0000", 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9575999999999998e-05 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518584+0000", 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0359e-05 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.524019+0000", 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005435151 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.524061+0000", 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1631000000000002e-05 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.0s1 54/31 ECSubWrite(tid=30, reqid=client.4615.0:421, at_version=0'0, trim_to=0'0, roll_forward_to=54'16))", 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.518829+0000", 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.9675577440000005, 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052386389999999998, 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.587 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518829+0000", 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518829+0000", 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518830+0000", 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5900000000000005e-07 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518840+0000", 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0241000000000001e-05 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518852+0000", 2024-09-16T03:40:43.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1989e-05 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518866+0000", 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3515e-05 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518938+0000", 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2094000000000005e-05 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518949+0000", 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0852999999999999e-05 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.524037+0000", 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050878080000000001 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.524068+0000", 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1180000000000003e-05 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.591 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.0s1 58/31 ECSubWrite(tid=54, reqid=client.4615.0:463, at_version=58'23, trim_to=0'0, roll_forward_to=58'20))", 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:37.484204+0000", 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.0021825629999999, 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052434250000000003, 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484204+0000", 2024-09-16T03:40:43.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484204+0000", 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484205+0000", 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1399999999999996e-07 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484222+0000", 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7232999999999999e-05 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484235+0000", 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2421e-05 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484244+0000", 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6879999999999994e-06 2024-09-16T03:40:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484290+0000", 2024-09-16T03:40:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5495999999999999e-05 2024-09-16T03:40:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484295+0000", 2024-09-16T03:40:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0690000000000001e-06 2024-09-16T03:40:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.489376+0000", 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050810489999999998 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.489448+0000", 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1755000000000007e-05 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:43.597 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:43.598 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:43.599 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:40:43.599 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:43.595+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:43.606 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:43.607 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:43.625 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:43.625 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:43.642 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:43.642 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:43.658 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:43.675 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:43.675 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:43.693 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:43.698 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:43.697+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:43.710 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:43.710 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:43.714 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:43.714 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:43.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:40:43.741 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:43.741 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:43.757 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:43.757 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:43.774 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:43.774 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:43.779 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /var/log/ceph/exp.3.fs1.13 2024-09-16T03:40:43.791 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:43.791 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:43.799 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:43.797+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:43.803 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.13 2024-09-16T03:40:43.804 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-13 show-label 2024-09-16T03:40:43.806 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:43.807 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:43.807 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-09-16T03:40:43.825 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:43.823+0000 7f27c37fe640 0 -- 172.21.15.76:0/2043990446 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f27c421ee00 msgr2=0x7f27c421f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:43.826 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:43.826 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:43.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:40:43.884 INFO:teuthology.orchestra.run.smithi103.stdout:inferring bluefs devices from bluestore path 2024-09-16T03:40:43.900 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:43.898+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:43.979 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:43.979 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:43.987 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:40:44.031 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:44.029+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:44.044 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:44.045 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:40:44.047 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:40:44.091 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:44.091 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:44.100 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:40:44.102 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:44.100+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:44.150 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "/var/lib/ceph/osd/ceph-13/block": { 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_uuid": "61e9204b-8269-4c35-b409-5cfbd5e6abbb", 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 96636764160, 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "btime": "2024-09-16T03:38:53.677314+0000", 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "main", 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "bfm_blocks": "23592960", 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "bfm_blocks_per_key": "128", 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "bfm_bytes_per_block": "4096", 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "bfm_size": "96636764160", 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs": "1", 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_fsid": "ecf7a42a-e941-4b24-8693-0ff8b6eb41b1", 2024-09-16T03:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version_when_created": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: "created_at": "2024-09-16T03:38:57.050911Z", 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: "elastic_shared_blobs": "1", 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: "epoch": "16", 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: "kv_backend": "rocksdb", 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: "magic": "ceph osd volume v026", 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: "multi": "yes", 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: "ready": "ready", 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: "require_osd_release": "19", 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bluestore", 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: "whoami": "13" 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:44.152 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:44.153 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.13 2024-09-16T03:40:44.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-13 show-sharding 2024-09-16T03:40:44.202 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:44.200+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:44.242 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:44.243 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:44.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:40:44.304 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:44.302+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:44.356 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:44.356 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:44.365 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:40:44.405 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:44.403+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:44.468 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:44.468 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:44.476 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:40:44.478 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T03:40:44.482 INFO:tasks.rados.rados.0.smithi121.stdout:189: done (0 left) 2024-09-16T03:40:44.482 INFO:tasks.rados.rados.0.smithi121.stdout:190: rollback oid 1 current snap is 20 2024-09-16T03:40:44.482 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 1 to 18 2024-09-16T03:40:44.482 INFO:tasks.rados.rados.0.smithi121.stdout:191: rollback oid 22 current snap is 20 2024-09-16T03:40:44.482 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 22 to 3 2024-09-16T03:40:44.482 INFO:tasks.rados.rados.0.smithi121.stdout:192: read oid 2 snap -1 2024-09-16T03:40:44.483 INFO:tasks.rados.rados.0.smithi121.stdout:192: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-16T03:40:44.483 INFO:tasks.rados.rados.0.smithi121.stdout:193: read oid 25 snap -1 2024-09-16T03:40:44.483 INFO:tasks.rados.rados.0.smithi121.stdout:193: expect (ObjNum 61 snap 2 seq_num 61) 2024-09-16T03:40:44.483 INFO:tasks.rados.rados.0.smithi121.stdout:194: read oid 30 snap -1 2024-09-16T03:40:44.483 INFO:tasks.rados.rados.0.smithi121.stdout:194: expect (ObjNum 81 snap 12 seq_num 81) 2024-09-16T03:40:44.483 INFO:tasks.rados.rados.0.smithi121.stdout:195: rollback oid 36 current snap is 20 2024-09-16T03:40:44.483 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 36 to 17 2024-09-16T03:40:44.483 INFO:tasks.rados.rados.0.smithi121.stdout:196: rmattr oid 29 current snap is 20 2024-09-16T03:40:44.483 INFO:tasks.rados.rados.0.smithi121.stdout:196: done (6 left) 2024-09-16T03:40:44.483 INFO:tasks.rados.rados.0.smithi121.stdout:197: delete oid 21 current snap is 20 2024-09-16T03:40:44.489 INFO:tasks.rados.rados.0.smithi121.stdout:191: finishing rollback tid 1 to smithi12138290-22 2024-09-16T03:40:44.489 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 22 v 0 (ObjNum 129 snap 0 seq_num 1107097296) dirty dne 2024-09-16T03:40:44.489 INFO:tasks.rados.rados.0.smithi121.stdout:191: done (6 left) 2024-09-16T03:40:44.489 INFO:tasks.rados.rados.0.smithi121.stdout:197: done (5 left) 2024-09-16T03:40:44.489 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 39 current snap is 20 2024-09-16T03:40:44.489 INFO:tasks.rados.rados.0.smithi121.stdout:198: seq_num 101 ranges {0=540672,540672=507904,1048576=622592,1671168=32768} 2024-09-16T03:40:44.492 INFO:tasks.rados.rados.0.smithi121.stdout:198: writing smithi12138290-39 from 0 to 540672 tid 1 2024-09-16T03:40:44.495 INFO:tasks.rados.rados.0.smithi121.stdout:198: writing smithi12138290-39 from 540672 to 1048576 tid 2 2024-09-16T03:40:44.499 INFO:tasks.rados.rados.0.smithi121.stdout:198: writing smithi12138290-39 from 1048576 to 1671168 tid 3 2024-09-16T03:40:44.500 INFO:tasks.rados.rados.0.smithi121.stdout:198: writing smithi12138290-39 from 1671168 to 1703936 tid 4 2024-09-16T03:40:44.500 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 17 current snap is 20 2024-09-16T03:40:44.500 INFO:tasks.rados.rados.0.smithi121.stdout:195: finishing rollback tid 1 to smithi12138290-36 2024-09-16T03:40:44.500 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 36 v 0 (ObjNum 129 snap 0 seq_num 1105042104) dirty dne 2024-09-16T03:40:44.500 INFO:tasks.rados.rados.0.smithi121.stdout:193: expect (ObjNum 61 snap 2 seq_num 61) 2024-09-16T03:40:44.505 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:44.503+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:44.543 INFO:tasks.rados.rados.0.smithi121.stdout:192: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-16T03:40:44.566 INFO:tasks.rados.rados.0.smithi121.stdout:198: finishing write tid 1 to smithi12138290-39 2024-09-16T03:40:44.566 INFO:tasks.rados.rados.0.smithi121.stdout:198: finishing write tid 2 to smithi12138290-39 2024-09-16T03:40:44.566 INFO:tasks.rados.rados.0.smithi121.stdout:190: finishing rollback tid 0 to smithi12138290-1 2024-09-16T03:40:44.566 INFO:tasks.rados.rados.0.smithi121.stdout:198: finishing write tid 3 to smithi12138290-39 2024-09-16T03:40:44.566 INFO:tasks.rados.rados.0.smithi121.stdout:198: finishing write tid 4 to smithi12138290-39 2024-09-16T03:40:44.566 INFO:tasks.rados.rados.0.smithi121.stdout:198: finishing write tid 5 to smithi12138290-39 2024-09-16T03:40:44.567 INFO:tasks.rados.rados.0.smithi121.stdout:194: expect (ObjNum 81 snap 12 seq_num 81) 2024-09-16T03:40:44.584 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:44.584 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:44.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:40:44.606 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:44.604+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:44.676 INFO:tasks.rados.rados.0.smithi121.stdout:190: finishing rollback tid 1 to smithi12138290-1 2024-09-16T03:40:44.676 INFO:tasks.rados.rados.0.smithi121.stdout:190: finishing rollback tid 2 to smithi12138290-1 2024-09-16T03:40:44.676 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 1 v 60 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:40:44.677 INFO:tasks.rados.rados.0.smithi121.stdout:198: finishing write tid 6 to smithi12138290-39 2024-09-16T03:40:44.677 INFO:tasks.rados.rados.0.smithi121.stdout:198: oid 39 updating version 0 to 32 2024-09-16T03:40:44.677 INFO:tasks.rados.rados.0.smithi121.stdout:198: oid 39 version 32 is already newer than 29 2024-09-16T03:40:44.677 INFO:tasks.rados.rados.0.smithi121.stdout:198: oid 39 version 32 is already newer than 30 2024-09-16T03:40:44.677 INFO:tasks.rados.rados.0.smithi121.stdout:198: oid 39 version 32 is already newer than 31 2024-09-16T03:40:44.677 INFO:tasks.rados.rados.0.smithi121.stdout:198: oid 39 version 32 is already newer than 28 2024-09-16T03:40:44.677 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 39 v 32 (ObjNum 101 snap 20 seq_num 101) dirty exists 2024-09-16T03:40:44.677 INFO:tasks.rados.rados.0.smithi121.stdout:198: left oid 39 (ObjNum 101 snap 20 seq_num 101) 2024-09-16T03:40:44.677 INFO:tasks.rados.rados.0.smithi121.stdout:199: seq_num 102 ranges {278528=557056,835584=638976,1474560=524288} 2024-09-16T03:40:44.680 INFO:tasks.rados.rados.0.smithi121.stdout:199: writing smithi12138290-17 from 278528 to 835584 tid 1 2024-09-16T03:40:44.683 INFO:tasks.rados.rados.0.smithi121.stdout:199: writing smithi12138290-17 from 835584 to 1474560 tid 2 2024-09-16T03:40:44.687 INFO:tasks.rados.rados.0.smithi121.stdout:199: writing smithi12138290-17 from 1474560 to 1998848 tid 3 2024-09-16T03:40:44.687 INFO:tasks.rados.rados.0.smithi121.stdout:190: done (6 left) 2024-09-16T03:40:44.687 INFO:tasks.rados.rados.0.smithi121.stdout:192: done (5 left) 2024-09-16T03:40:44.687 INFO:tasks.rados.rados.0.smithi121.stdout:193: done (4 left) 2024-09-16T03:40:44.687 INFO:tasks.rados.rados.0.smithi121.stdout:194: done (3 left) 2024-09-16T03:40:44.687 INFO:tasks.rados.rados.0.smithi121.stdout:195: done (2 left) 2024-09-16T03:40:44.687 INFO:tasks.rados.rados.0.smithi121.stdout:198: done (1 left) 2024-09-16T03:40:44.687 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 34 current snap is 20 2024-09-16T03:40:44.687 INFO:tasks.rados.rados.0.smithi121.stdout:200: seq_num 103 ranges {1671168=573440,2244608=786432,3031040=376832} 2024-09-16T03:40:44.698 INFO:tasks.rados.rados.0.smithi121.stdout:200: writing smithi12138290-34 from 1671168 to 2244608 tid 1 2024-09-16T03:40:44.698 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:44.698 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:44.702 INFO:tasks.rados.rados.0.smithi121.stdout:200: writing smithi12138290-34 from 2244608 to 3031040 tid 2 2024-09-16T03:40:44.705 INFO:tasks.rados.rados.0.smithi121.stdout:200: writing smithi12138290-34 from 3031040 to 3407872 tid 3 2024-09-16T03:40:44.705 INFO:tasks.rados.rados.0.smithi121.stdout:201: delete oid 21 current snap is 20 2024-09-16T03:40:44.705 INFO:tasks.rados.rados.0.smithi121.stdout:199: finishing write tid 1 to smithi12138290-17 2024-09-16T03:40:44.705 INFO:tasks.rados.rados.0.smithi121.stdout:199: finishing write tid 2 to smithi12138290-17 2024-09-16T03:40:44.705 INFO:tasks.rados.rados.0.smithi121.stdout:199: finishing write tid 3 to smithi12138290-17 2024-09-16T03:40:44.705 INFO:tasks.rados.rados.0.smithi121.stdout:199: finishing write tid 4 to smithi12138290-17 2024-09-16T03:40:44.706 INFO:tasks.rados.rados.0.smithi121.stdout:199: finishing write tid 5 to smithi12138290-17 2024-09-16T03:40:44.706 INFO:tasks.rados.rados.0.smithi121.stdout:199: oid 17 updating version 0 to 61 2024-09-16T03:40:44.706 INFO:tasks.rados.rados.0.smithi121.stdout:199: oid 17 version 61 is already newer than 58 2024-09-16T03:40:44.706 INFO:tasks.rados.rados.0.smithi121.stdout:199: oid 17 version 61 is already newer than 59 2024-09-16T03:40:44.706 INFO:tasks.rados.rados.0.smithi121.stdout:199: oid 17 version 61 is already newer than 60 2024-09-16T03:40:44.706 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 17 v 61 (ObjNum 102 snap 20 seq_num 102) dirty exists 2024-09-16T03:40:44.706 INFO:tasks.rados.rados.0.smithi121.stdout:199: left oid 17 (ObjNum 102 snap 20 seq_num 102) 2024-09-16T03:40:44.707 INFO:tasks.rados.rados.0.smithi121.stdout:199: done (2 left) 2024-09-16T03:40:44.707 INFO:tasks.rados.rados.0.smithi121.stdout:201: done (1 left) 2024-09-16T03:40:44.707 INFO:tasks.rados.rados.0.smithi121.stdout:202: read oid 8 snap -1 2024-09-16T03:40:44.707 INFO:tasks.rados.rados.0.smithi121.stdout:202: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:40:44.707 INFO:tasks.rados.rados.0.smithi121.stdout:203: snap_remove snap 18 2024-09-16T03:40:44.708 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:44.705+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:44.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:40:44.808 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:44.806+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:44.811 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:44.811 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:44.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:40:44.909 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:44.907+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:44.962 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:44.962 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:44.971 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:40:45.010 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:45.008+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:45.073 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:45.073 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:45.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:40:45.112 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:45.109+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:45.185 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:45.185 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:45.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:40:45.212 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:45.211+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:45.274 INFO:teuthology.orchestra.run.smithi103.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-09-16T03:40:45.278 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.13 2024-09-16T03:40:45.278 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-09-16T03:40:45.278 INFO:tasks.thrashosds.thrasher:applying new sharding: m(3) O L(4,5-) 2024-09-16T03:40:45.278 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-13 --sharding 'm(3) O L(4,5-)' reshard 2024-09-16T03:40:45.296 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:45.296 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:45.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:40:45.313 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:45.311+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:45.407 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:45.408 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:45.414 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:45.412+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:45.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:40:45.490 INFO:tasks.rados.rados.0.smithi121.stdout:203: done (2 left) 2024-09-16T03:40:45.490 INFO:tasks.rados.rados.0.smithi121.stdout:204: copy_from oid 50 from oid 42 current snap is 20 2024-09-16T03:40:45.490 INFO:tasks.rados.rados.0.smithi121.stdout:200: finishing write tid 1 to smithi12138290-34 2024-09-16T03:40:45.490 INFO:tasks.rados.rados.0.smithi121.stdout:200: finishing write tid 2 to smithi12138290-34 2024-09-16T03:40:45.490 INFO:tasks.rados.rados.0.smithi121.stdout:200: finishing write tid 3 to smithi12138290-34 2024-09-16T03:40:45.490 INFO:tasks.rados.rados.0.smithi121.stdout:202: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:40:45.497 INFO:tasks.rados.rados.0.smithi121.stdout:200: finishing write tid 4 to smithi12138290-34 2024-09-16T03:40:45.497 INFO:tasks.rados.rados.0.smithi121.stdout:200: finishing write tid 5 to smithi12138290-34 2024-09-16T03:40:45.497 INFO:tasks.rados.rados.0.smithi121.stdout:200: oid 34 updating version 0 to 26 2024-09-16T03:40:45.497 INFO:tasks.rados.rados.0.smithi121.stdout:200: oid 34 updating version 26 to 28 2024-09-16T03:40:45.497 INFO:tasks.rados.rados.0.smithi121.stdout:200: oid 34 version 28 is already newer than 27 2024-09-16T03:40:45.497 INFO:tasks.rados.rados.0.smithi121.stdout:200: oid 34 updating version 28 to 29 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 34 v 29 (ObjNum 103 snap 20 seq_num 103) dirty exists 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:200: left oid 34 (ObjNum 103 snap 20 seq_num 103) 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:200: done (2 left) 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:202: done (1 left) 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:205: rollback oid 46 current snap is 20 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 46 to 16 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:206: copy_from oid 49 from oid 14 current snap is 20 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:207: read oid 3 snap 3 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:207: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:208: read oid 41 snap -1 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:208: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:209: copy_from oid 7 from oid 18 current snap is 20 2024-09-16T03:40:45.498 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 40 current snap is 20 2024-09-16T03:40:45.499 INFO:tasks.rados.rados.0.smithi121.stdout:210: seq_num 105 ranges {0=606208,606208=688128,1294336=475136} 2024-09-16T03:40:45.500 INFO:tasks.rados.rados.0.smithi121.stdout:210: writing smithi12138290-40 from 0 to 606208 tid 1 2024-09-16T03:40:45.506 INFO:tasks.rados.rados.0.smithi121.stdout:210: writing smithi12138290-40 from 606208 to 1294336 tid 2 2024-09-16T03:40:45.509 INFO:tasks.rados.rados.0.smithi121.stdout:210: writing smithi12138290-40 from 1294336 to 1769472 tid 3 2024-09-16T03:40:45.509 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 21 current snap is 20 2024-09-16T03:40:45.510 INFO:tasks.rados.rados.0.smithi121.stdout:211: seq_num 106 ranges {0=622592,622592=442368,1064960=524288,1589248=196608} 2024-09-16T03:40:45.515 INFO:tasks.rados.rados.0.smithi121.stdout:211: writing smithi12138290-21 from 0 to 622592 tid 1 2024-09-16T03:40:45.515 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:45.513+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:45.517 INFO:tasks.rados.rados.0.smithi121.stdout:211: writing smithi12138290-21 from 622592 to 1064960 tid 2 2024-09-16T03:40:45.521 INFO:tasks.rados.rados.0.smithi121.stdout:211: writing smithi12138290-21 from 1064960 to 1589248 tid 3 2024-09-16T03:40:45.523 INFO:tasks.rados.rados.0.smithi121.stdout:211: writing smithi12138290-21 from 1589248 to 1785856 tid 4 2024-09-16T03:40:45.524 INFO:tasks.rados.rados.0.smithi121.stdout:212: read oid 6 snap -1 2024-09-16T03:40:45.524 INFO:tasks.rados.rados.0.smithi121.stdout:212: expect (ObjNum 57 snap 0 seq_num 57) 2024-09-16T03:40:45.524 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 33 current snap is 20 2024-09-16T03:40:45.524 INFO:tasks.rados.rados.0.smithi121.stdout:213: seq_num 107 ranges {802816=638976,1441792=458752,1900544=704512} 2024-09-16T03:40:45.532 INFO:tasks.rados.rados.0.smithi121.stdout:213: writing smithi12138290-33 from 802816 to 1441792 tid 1 2024-09-16T03:40:45.536 INFO:tasks.rados.rados.0.smithi121.stdout:213: writing smithi12138290-33 from 1441792 to 1900544 tid 2 2024-09-16T03:40:45.541 INFO:tasks.rados.rados.0.smithi121.stdout:213: writing smithi12138290-33 from 1900544 to 2605056 tid 3 2024-09-16T03:40:45.542 INFO:tasks.rados.rados.0.smithi121.stdout:214: copy_from oid 48 from oid 13 current snap is 20 2024-09-16T03:40:45.542 INFO:tasks.rados.rados.0.smithi121.stdout:207: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-16T03:40:45.544 INFO:tasks.rados.rados.0.smithi121.stdout:204: finishing copy_from racing read to smithi12138290-50 2024-09-16T03:40:45.544 INFO:tasks.rados.rados.0.smithi121.stdout:205: finishing rollback tid 0 to smithi12138290-46 2024-09-16T03:40:45.544 INFO:tasks.rados.rados.0.smithi121.stdout:206: finishing copy_from racing read to smithi12138290-49 2024-09-16T03:40:45.544 INFO:tasks.rados.rados.0.smithi121.stdout:207: done (10 left) 2024-09-16T03:40:45.545 INFO:tasks.rados.rados.0.smithi121.stdout:215: read oid 38 snap -1 2024-09-16T03:40:45.545 INFO:tasks.rados.rados.0.smithi121.stdout:215: expect deleted 2024-09-16T03:40:45.545 INFO:tasks.rados.rados.0.smithi121.stdout:216: copy_from oid 34 from oid 32 current snap is 20 2024-09-16T03:40:45.545 INFO:tasks.rados.rados.0.smithi121.stdout:205: finishing rollback tid 1 to smithi12138290-46 2024-09-16T03:40:45.545 INFO:tasks.rados.rados.0.smithi121.stdout:205: finishing rollback tid 2 to smithi12138290-46 2024-09-16T03:40:45.545 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 46 v 29 (ObjNum 87 snap 12 seq_num 87) dirty exists 2024-09-16T03:40:45.545 INFO:tasks.rados.rados.0.smithi121.stdout:209: finishing copy_from racing read to smithi12138290-7 2024-09-16T03:40:45.545 INFO:tasks.rados.rados.0.smithi121.stdout:204: finishing copy_from to smithi12138290-50 2024-09-16T03:40:45.545 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 50 v 4 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:40:45.545 INFO:tasks.rados.rados.0.smithi121.stdout:208: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:40:45.545 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:45.545 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:45.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:40:45.564 INFO:tasks.rados.rados.0.smithi121.stdout:210: finishing write tid 1 to smithi12138290-40 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:206: finishing copy_from to smithi12138290-49 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 49 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:210: finishing write tid 2 to smithi12138290-40 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:210: finishing write tid 3 to smithi12138290-40 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:209: finishing copy_from to smithi12138290-7 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 7 v 48 (ObjNum 89 snap 12 seq_num 89) dirty exists 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:210: finishing write tid 4 to smithi12138290-40 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:211: finishing write tid 1 to smithi12138290-21 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:210: finishing write tid 5 to smithi12138290-40 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:210: oid 40 updating version 0 to 28 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:210: oid 40 version 28 is already newer than 25 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:210: oid 40 updating version 28 to 29 2024-09-16T03:40:45.565 INFO:tasks.rados.rados.0.smithi121.stdout:210: oid 40 version 29 is already newer than 27 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 40 v 29 (ObjNum 105 snap 20 seq_num 105) dirty exists 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:210: left oid 40 (ObjNum 105 snap 20 seq_num 105) 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:211: finishing write tid 2 to smithi12138290-21 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:204: done (11 left) 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:205: done (10 left) 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:206: done (9 left) 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:208: done (8 left) 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:209: done (7 left) 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:210: done (6 left) 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:217: read oid 30 snap 16 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:217: expect (ObjNum 81 snap 12 seq_num 81) 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:218: snap_create 2024-09-16T03:40:45.566 INFO:tasks.rados.rados.0.smithi121.stdout:211: finishing write tid 3 to smithi12138290-21 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:211: finishing write tid 4 to smithi12138290-21 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:211: finishing write tid 5 to smithi12138290-21 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:211: finishing write tid 6 to smithi12138290-21 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:211: oid 21 updating version 0 to 19 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:211: oid 21 version 19 is already newer than 17 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:211: oid 21 version 19 is already newer than 16 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:211: oid 21 version 19 is already newer than 15 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:211: oid 21 version 19 is already newer than 18 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 21 v 19 (ObjNum 106 snap 20 seq_num 106) dirty exists 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:211: left oid 21 (ObjNum 106 snap 20 seq_num 106) 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:213: finishing write tid 1 to smithi12138290-33 2024-09-16T03:40:45.567 INFO:tasks.rados.rados.0.smithi121.stdout:212: expect (ObjNum 57 snap 0 seq_num 57) 2024-09-16T03:40:45.593 INFO:tasks.rados.rados.0.smithi121.stdout:213: finishing write tid 2 to smithi12138290-33 2024-09-16T03:40:45.593 INFO:tasks.rados.rados.0.smithi121.stdout:214: finishing copy_from to smithi12138290-48 2024-09-16T03:40:45.593 INFO:tasks.rados.rados.0.smithi121.stdout:214: got expected ENOENT (src dne) 2024-09-16T03:40:45.593 INFO:tasks.rados.rados.0.smithi121.stdout:214: finishing copy_from racing read to smithi12138290-48 2024-09-16T03:40:45.593 INFO:tasks.rados.rados.0.smithi121.stdout:213: finishing write tid 3 to smithi12138290-33 2024-09-16T03:40:45.593 INFO:tasks.rados.rados.0.smithi121.stdout:216: finishing copy_from to smithi12138290-34 2024-09-16T03:40:45.594 INFO:tasks.rados.rados.0.smithi121.stdout:216: got expected ENOENT (src dne) 2024-09-16T03:40:45.594 INFO:tasks.rados.rados.0.smithi121.stdout:216: finishing copy_from racing read to smithi12138290-34 2024-09-16T03:40:45.594 INFO:tasks.rados.rados.0.smithi121.stdout:213: finishing write tid 4 to smithi12138290-33 2024-09-16T03:40:45.594 INFO:tasks.rados.rados.0.smithi121.stdout:213: finishing write tid 5 to smithi12138290-33 2024-09-16T03:40:45.594 INFO:tasks.rados.rados.0.smithi121.stdout:213: oid 33 updating version 0 to 31 2024-09-16T03:40:45.594 INFO:tasks.rados.rados.0.smithi121.stdout:213: oid 33 version 31 is already newer than 29 2024-09-16T03:40:45.594 INFO:tasks.rados.rados.0.smithi121.stdout:213: oid 33 updating version 31 to 32 2024-09-16T03:40:45.594 INFO:tasks.rados.rados.0.smithi121.stdout:213: oid 33 version 32 is already newer than 30 2024-09-16T03:40:45.595 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 33 v 32 (ObjNum 107 snap 20 seq_num 107) dirty exists 2024-09-16T03:40:45.595 INFO:tasks.rados.rados.0.smithi121.stdout:213: left oid 33 (ObjNum 107 snap 20 seq_num 107) 2024-09-16T03:40:45.595 INFO:tasks.rados.rados.0.smithi121.stdout:217: expect (ObjNum 81 snap 12 seq_num 81) 2024-09-16T03:40:45.616 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:45.614+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:45.660 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:45.660 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:45.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:40:45.717 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:45.715+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:45.811 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:45.811 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:45.818 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:45.817+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:45.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:40:45.919 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:45.917+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:45.960 INFO:teuthology.orchestra.run.smithi103.stdout:reshard success 2024-09-16T03:40:45.963 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:45.963 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:45.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:40:46.020 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:46.018+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:46.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:46.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:40:46.125 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:46.124+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:46.184 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:46.184 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:46.185 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:46.185 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:46.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:40:46.202 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:46.202 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:46.219 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:46.219 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:46.222 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:46.221+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:46.235 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:46.235 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:46.251 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:46.268 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:46.284 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:46.285 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:46.301 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:46.301 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:46.317 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:46.317 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:46.323 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:46.321+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:46.333 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:46.336 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:46.336 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:46.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:40:46.396 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.13 2024-09-16T03:40:46.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-13 fsck 2024-09-16T03:40:46.424 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:46.422+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:46.447 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:46.447 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:46.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:40:46.476 INFO:tasks.rados.rados.0.smithi121.stdout:211: done (7 left) 2024-09-16T03:40:46.476 INFO:tasks.rados.rados.0.smithi121.stdout:212: done (6 left) 2024-09-16T03:40:46.476 INFO:tasks.rados.rados.0.smithi121.stdout:213: done (5 left) 2024-09-16T03:40:46.476 INFO:tasks.rados.rados.0.smithi121.stdout:214: done (4 left) 2024-09-16T03:40:46.476 INFO:tasks.rados.rados.0.smithi121.stdout:215: done (3 left) 2024-09-16T03:40:46.476 INFO:tasks.rados.rados.0.smithi121.stdout:216: done (2 left) 2024-09-16T03:40:46.476 INFO:tasks.rados.rados.0.smithi121.stdout:217: done (1 left) 2024-09-16T03:40:46.477 INFO:tasks.rados.rados.0.smithi121.stdout:218: done (0 left) 2024-09-16T03:40:46.477 INFO:tasks.rados.rados.0.smithi121.stdout:219: read oid 28 snap -1 2024-09-16T03:40:46.477 INFO:tasks.rados.rados.0.smithi121.stdout:219: expect (ObjNum 73 snap 6 seq_num 73) 2024-09-16T03:40:46.477 INFO:tasks.rados.rados.0.smithi121.stdout:220: read oid 41 snap -1 2024-09-16T03:40:46.477 INFO:tasks.rados.rados.0.smithi121.stdout:220: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:40:46.477 INFO:tasks.rados.rados.0.smithi121.stdout:221: read oid 42 snap -1 2024-09-16T03:40:46.477 INFO:tasks.rados.rados.0.smithi121.stdout:221: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:40:46.477 INFO:tasks.rados.rados.0.smithi121.stdout:222: snap_create 2024-09-16T03:40:46.480 INFO:tasks.rados.rados.0.smithi121.stdout:221: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:40:46.484 INFO:tasks.rados.rados.0.smithi121.stdout:220: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:40:46.501 INFO:tasks.rados.rados.0.smithi121.stdout:219: expect (ObjNum 73 snap 6 seq_num 73) 2024-09-16T03:40:46.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:40:46.524 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:46.522+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:46.553 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:46.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:46.569 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:46.569 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:46.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:40:46.585 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-09-16T03:40:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:46.584+0000 7fd1677fe640 0 -- 172.21.15.76:0/662049805 >> [v2:172.21.15.119:6800/1147733023,v1:172.21.15.119:6801/1147733023] conn(0x7fd168218f30 msgr2=0x7fd1682193c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:46.625 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:46.623+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:46.678 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:46.678 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:46.678 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:46.678 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:46.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:40:46.726 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:46.724+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:46.804 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:46.821 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:46.821 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:40:46.825 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:40:46.826 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:46.825+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:46.833 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:46.833 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:46.833 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:46.834 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:46.834 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:46.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:40:46.927 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:46.925+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:46.990 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:46.990 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:46.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:46.990 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:46.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.990 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:225 3.fs0 3:fe9db9b8:::smithi12138290-24:head [append 0~720896] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:46.990 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.538459+0000", 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 29.448829104000001, 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076796640000000001, 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 225 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.538459+0000", 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.538459+0000", 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.538462+0000", 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2990000000000001e-06 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539454+0000", 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00099185099999999997 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539477+0000", 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3056000000000001e-05 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539487+0000", 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0019e-05 2024-09-16T03:40:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539564+0000", 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6762999999999997e-05 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.539570+0000", 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5359999999999998e-06 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:46.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541406+0000", 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001836086 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541468+0000", 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1659e-05 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542536+0000", 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001067597 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545013+0000", 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002477061 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.546056+0000", 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00104386 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.546138+0000", 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1877000000000002e-05 2024-09-16T03:40:46.996 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:226 3.fs0 3:fe9db9b8:::smithi12138290-24:head [append 0~376832] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 29.446833672, 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095498700000000002, 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:46.997 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 226 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:46.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540455+0000", 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540861+0000", 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040549800000000001 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:46.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540874+0000", 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2974e-05 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540882+0000", 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540948+0000", 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6383999999999993e-05 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540954+0000", 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9730000000000002e-06 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543210+0000", 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022561209999999998 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543290+0000", 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9907999999999995e-05 2024-09-16T03:40:47.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543922+0000", 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000631713 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545016+0000", 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001093896 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.549959+0000", 2024-09-16T03:40:47.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049431980000000002 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550004+0000", 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4858999999999999e-05 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:227 3.fs0 3:fe9db9b8:::smithi12138290-24:head [setxattr _header (56) in=7b] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 29.44638003, 2024-09-16T03:40:47.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092805639999999998, 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 227 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:47.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540909+0000", 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6400000000000001e-07 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540912+0000", 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1439999999999999e-06 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540919+0000", 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4000000000000003e-06 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540925+0000", 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6069999999999998e-06 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541006+0000", 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1231999999999997e-05 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541012+0000", 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3609999999999997e-06 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544371+0000", 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033583739999999999 2024-09-16T03:40:47.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544424+0000", 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3535000000000002e-05 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544709+0000", 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028452099999999998 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545291+0000", 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00058281499999999998 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550110+0000", 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048181430000000004 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550188+0000", 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8668e-05 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.009 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:228 3.fs0 3:fe9db9b8:::smithi12138290-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 29.446338875999999, 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011210417, 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.010 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 228 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0399999999999996e-07 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540952+0000", 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6860000000000002e-06 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540960+0000", 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9729999999999997e-06 2024-09-16T03:40:47.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540966+0000", 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7509999999999997e-06 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541041+0000", 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5216999999999995e-05 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541046+0000", 2024-09-16T03:40:47.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0000000000000004e-06 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544875+0000", 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038287339999999999 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544964+0000", 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9245000000000004e-05 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550540+0000", 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055763649999999998 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550618+0000", 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7908e-05 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552159+0000", 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015409340000000001 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.015 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:345 3.fs0 3:fe9db9b8:::smithi12138290-24:head [copy-from ver 4] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:31.497360+0000", 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.489927695, 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013644073, 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.016 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 345 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497360+0000", 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497360+0000", 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497362+0000", 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.598e-06 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497371+0000", 2024-09-16T03:40:47.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0180000000000007e-06 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497382+0000", 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1562e-05 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497389+0000", 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4409999999999998e-06 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497446+0000", 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7054e-05 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497451+0000", 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.339e-06 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499316+0000", 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001864517 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499415+0000", 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9077e-05 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.505946+0000", 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006531294 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.507163+0000", 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001216649 2024-09-16T03:40:47.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.510932+0000", 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037689820000000001 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.511004+0000", 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2541999999999998e-05 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:346 3.fs0 3:fe9db9b8:::smithi12138290-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.489883185, 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096185339999999998, 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 346 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:40:47.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0499999999999999e-07 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497407+0000", 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497412+0000", 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3859999999999998e-06 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497415+0000", 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1439999999999999e-06 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497492+0000", 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6577999999999998e-05 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497497+0000", 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.023e-06 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499594+0000", 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020968380000000002 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499632+0000", 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8133000000000002e-05 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.506768+0000", 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071363449999999997 2024-09-16T03:40:47.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.506883+0000", 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011502000000000001 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.507023+0000", 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013991399999999999 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds3 53/31 ECSubWrite(tid=27, reqid=client.4615.0:354, at_version=53'8, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.503762+0000", 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.483526231000001, 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096166930000000008, 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503762+0000", 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503762+0000", 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503762+0000", 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0500000000000004e-07 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503788+0000", 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5888999999999999e-05 2024-09-16T03:40:47.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503798+0000", 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6849999999999996e-06 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503806+0000", 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1850000000000004e-06 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503878+0000", 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2565999999999996e-05 2024-09-16T03:40:47.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503888+0000", 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7880000000000001e-06 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513344+0000", 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0094557970000000002 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513378+0000", 2024-09-16T03:40:47.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4277999999999999e-05 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds3 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.505226+0000", 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.482061441999999, 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0081836519999999996, 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.033 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505226+0000", 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505226+0000", 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505227+0000", 2024-09-16T03:40:47.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0699999999999999e-06 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505249+0000", 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1469000000000001e-05 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505261+0000", 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2383e-05 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505269+0000", 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8169999999999999e-06 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506841+0000", 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001571624 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506851+0000", 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8179999999999999e-06 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513390+0000", 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065394889999999999 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513410+0000", 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9981999999999999e-05 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds3 53/31 ECSubWrite(tid=29, reqid=client.4615.0:356, at_version=53'10, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.505836+0000", 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.481452255000001, 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075990249999999997, 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505836+0000", 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505836+0000", 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505836+0000", 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2399999999999996e-07 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505854+0000", 2024-09-16T03:40:47.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7751e-05 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505869+0000", 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4834000000000001e-05 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505878+0000", 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5150000000000006e-06 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511244+0000", 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005366188 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511254+0000", 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0338e-05 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513412+0000", 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021583779999999999 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513435+0000", 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2096999999999999e-05 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:382 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.542264+0000", 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.445023751000001, 2024-09-16T03:40:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088709159999999995, 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 382 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542264+0000", 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542264+0000", 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542266+0000", 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.519e-06 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542272+0000", 2024-09-16T03:40:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0560000000000001e-06 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542283+0000", 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1265e-05 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542289+0000", 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9399999999999999e-06 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542345+0000", 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6131000000000002e-05 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542428+0000", 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3436999999999994e-05 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551135+0000", 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087065679999999996 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:383 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.444983269, 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010993826999999999, 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 383 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542308+0000", 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6450000000000001e-06 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542313+0000", 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2970000000000003e-06 2024-09-16T03:40:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542317+0000", 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5979999999999999e-06 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543899+0000", 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015818539999999999 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544026+0000", 2024-09-16T03:40:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012688099999999999 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.553298+0000", 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092721520000000005 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:384 3.fs0 3.1d9db97f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.542335+0000", 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.44495253, 2024-09-16T03:40:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0074699930000000003, 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 384 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542335+0000", 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542335+0000", 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542336+0000", 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9200000000000002e-07 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542338+0000", 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.599e-06 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542343+0000", 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2869999999999998e-06 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542346+0000", 2024-09-16T03:40:47.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0599999999999999e-06 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.549787+0000", 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0074413109999999999 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.549805+0000", 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8343999999999999e-05 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:382 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.550355+0000", 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.436932858, 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090719059999999994, 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 382 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.056 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550355+0000", 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550355+0000", 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550356+0000", 2024-09-16T03:40:47.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3699999999999999e-07 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550361+0000", 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7759999999999997e-06 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550369+0000", 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3639999999999999e-06 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550376+0000", 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5440000000000001e-06 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.553482+0000", 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031052829999999999 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.553615+0000", 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013317099999999999 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.559427+0000", 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058119310000000002 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:383 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.436897963, 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010309801, 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:47.060 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 383 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:40:47.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.77e-07 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550392+0000", 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0490000000000002e-06 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550397+0000", 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9699999999999998e-06 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550401+0000", 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4039999999999999e-06 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.553954+0000", 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003552955 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554020+0000", 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6389000000000004e-05 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560700+0000", 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066796570000000003 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:384 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:47.064 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.436882672999999, 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011709803, 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 384 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7399999999999999e-07 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550408+0000", 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.526e-06 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550412+0000", 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.408e-06 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550415+0000", 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8260000000000001e-06 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554733+0000", 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004318087 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554837+0000", 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010340699999999999 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.562115+0000", 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072782749999999998 2024-09-16T03:40:47.068 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:388 3.cs0 3:38c596e1:::smithi12138290-26:head [append 0~688128] snapc d=[d,b,8,5,3] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.550420+0000", 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.436867919999999, 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075429249999999998, 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.069 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 388 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550420+0000", 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550420+0000", 2024-09-16T03:40:47.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550420+0000", 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4400000000000001e-07 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551418+0000", 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00099834800000000003 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551431+0000", 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2121e-05 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551439+0000", 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0709999999999995e-06 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551505+0000", 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6859000000000005e-05 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551626+0000", 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000120028 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552697+0000", 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010710050000000001 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.557743+0000", 2024-09-16T03:40:47.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050463139999999997 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.557849+0000", 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000105976 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.557963+0000", 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000113959 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.074 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:390 3.cs0 3:38c596e1:::smithi12138290-26:head [append 0~229376] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.552611+0000", 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.434676602, 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076453199999999997, 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.075 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 390 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552611+0000", 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552611+0000", 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552612+0000", 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1999999999999998e-07 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552744+0000", 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013220500000000001 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552754+0000", 2024-09-16T03:40:47.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5650000000000007e-06 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552761+0000", 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554958+0000", 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021970380000000001 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.555069+0000", 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011048399999999999 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.555658+0000", 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00058942399999999996 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.557983+0000", 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002324544 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560205+0000", 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022224100000000002 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560256+0000", 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1513e-05 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.080 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:391 3.cs0 3:38c596e1:::smithi12138290-26:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.552773+0000", 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.434514647, 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076674600000000001, 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 391 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552773+0000", 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552773+0000", 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552774+0000", 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-07 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552790+0000", 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6271999999999999e-05 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552800+0000", 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5529999999999998e-06 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552805+0000", 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6269999999999997e-06 2024-09-16T03:40:47.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.556420+0000", 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036148640000000002 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.556511+0000", 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0517000000000001e-05 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.556923+0000", 2024-09-16T03:40:47.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00041256000000000002 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.558028+0000", 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011049549999999999 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560379+0000", 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002351021 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560441+0000", 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1432000000000004e-05 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:392 3.cs0 3:38c596e1:::smithi12138290-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.552815+0000", 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.434472793999999, 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091352450000000002, 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:47.086 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 392 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552815+0000", 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552815+0000", 2024-09-16T03:40:47.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552816+0000", 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9100000000000004e-07 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552818+0000", 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4679999999999999e-06 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552827+0000", 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8999999999999995e-06 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552834+0000", 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8870000000000003e-06 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.557200+0000", 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043658270000000001 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.557351+0000", 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000151722 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560770+0000", 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034184699999999998 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560830+0000", 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0352000000000001e-05 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.561950+0000", 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011200279999999999 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:469 3.cs0 3:38c596e1:::smithi12138290-26:head [copy-from ver 10] snapc 11=[11,10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:38.482894+0000", 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5043932949999999, 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0094610879999999994, 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.091 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 469 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482894+0000", 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482894+0000", 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482896+0000", 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8509999999999999e-06 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482903+0000", 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5479999999999998e-06 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482915+0000", 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2452999999999999e-05 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482924+0000", 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7390000000000005e-06 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482994+0000", 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0298999999999999e-05 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483098+0000", 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103646 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.487565+0000", 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044667689999999998 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.489179+0000", 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001614164 2024-09-16T03:40:47.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492105+0000", 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029258959999999999 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492356+0000", 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000250723 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.096 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.097 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:47.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:40:47.098 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:47.096+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:47.199 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:47.197+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:47.202 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:47.202 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:47.202 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:47.202 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:47.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:40:47.300 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:47.298+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:47.315 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:47.315 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:47.315 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:47.316 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:47.316 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:47.325 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:40:47.401 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:47.399+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:47.429 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.0s2 52/31 ECSubWrite(tid=24, reqid=client.4615.0:333, at_version=52'13, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:31.479901+0000", 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.947308854999999, 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038022300000000002, 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.479901+0000", 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.479901+0000", 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.479902+0000", 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8199999999999998e-07 2024-09-16T03:40:47.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.480446+0000", 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054430600000000004 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.480453+0000", 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.657e-06 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.480458+0000", 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2510000000000002e-06 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.480492+0000", 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4650000000000002e-05 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.480498+0000", 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4650000000000002e-06 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.483675+0000", 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031771999999999998 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.483703+0000", 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7818999999999999e-05 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds1 53/31 ECSubWrite(tid=27, reqid=client.4615.0:354, at_version=53'8, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.434 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.503897+0000", 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.923312219, 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046849650000000001, 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503897+0000", 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503897+0000", 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503898+0000", 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7300000000000002e-07 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503924+0000", 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6699999999999998e-05 2024-09-16T03:40:47.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503935+0000", 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0706999999999999e-05 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503944+0000", 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8829999999999995e-06 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504020+0000", 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5883000000000005e-05 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504030+0000", 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0532e-05 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508462+0000", 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044313699999999996 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508582+0000", 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000120517 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds1 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.504797+0000", 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.9224123, 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050530510000000002, 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.439 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504797+0000", 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504797+0000", 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504798+0000", 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1399999999999996e-07 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504822+0000", 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3915999999999999e-05 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504837+0000", 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5349000000000002e-05 2024-09-16T03:40:47.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504847+0000", 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5570000000000001e-06 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504905+0000", 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8418000000000002e-05 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504915+0000", 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8630000000000003e-06 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509808+0000", 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004892693 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509850+0000", 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2441000000000002e-05 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.443 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds1 53/31 ECSubWrite(tid=29, reqid=client.4615.0:356, at_version=53'10, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.506008+0000", 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.921201794, 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003866721, 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506008+0000", 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506008+0000", 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506009+0000", 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8899999999999998e-07 2024-09-16T03:40:47.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506027+0000", 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7880000000000002e-05 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506046+0000", 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9769999999999999e-05 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506053+0000", 2024-09-16T03:40:47.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1770000000000001e-06 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506859+0000", 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00080594200000000003 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506871+0000", 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1164e-05 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509855+0000", 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029848629999999999 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509874+0000", 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9035999999999999e-05 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds1 53/31 ECSubWrite(tid=31, reqid=client.4615.0:358, at_version=53'13, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.448 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.512639+0000", 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.914570249000001, 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057755219999999999, 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512639+0000", 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512639+0000", 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512640+0000", 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6799999999999999e-07 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512738+0000", 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7774000000000003e-05 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512757+0000", 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8581999999999999e-05 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512766+0000", 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6190000000000006e-06 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512841+0000", 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5031999999999996e-05 2024-09-16T03:40:47.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512854+0000", 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2965e-05 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518379+0000", 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055243920000000004 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518415+0000", 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.629e-05 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds1 53/31 ECSubWrite(tid=32, reqid=client.4615.0:359, at_version=53'14, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.513665+0000", 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.913544257, 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047721910000000003, 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.453 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513665+0000", 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513665+0000", 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513666+0000", 2024-09-16T03:40:47.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9700000000000005e-07 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513743+0000", 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6793999999999997e-05 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513756+0000", 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3455000000000001e-05 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513765+0000", 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7499999999999992e-06 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513850+0000", 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4523000000000004e-05 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513861+0000", 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1272e-05 2024-09-16T03:40:47.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518419+0000", 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045575249999999998 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518437+0000", 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8975e-05 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 53/31 ECSubWrite(tid=52, reqid=osd.10.0:51, at_version=0'0, trim_to=0'0, roll_forward_to=53'24))", 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.514274+0000", 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.912935466, 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041995239999999996, 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514274+0000", 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514274+0000", 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514275+0000", 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0410000000000001e-06 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514290+0000", 2024-09-16T03:40:47.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4513e-05 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514307+0000", 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7544e-05 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514317+0000", 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7059999999999999e-06 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514382+0000", 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4638e-05 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514391+0000", 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.346e-06 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518451+0000", 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040596870000000002 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518474+0000", 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3048999999999999e-05 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.9s4 53/31 ECSubWrite(tid=42, reqid=client.4615.0:362, at_version=53'21, trim_to=0'0, roll_forward_to=53'20))", 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.515405+0000", 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.911804611000001, 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060553209999999998, 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.462 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515405+0000", 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515405+0000", 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515406+0000", 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2600000000000001e-07 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515953+0000", 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054688600000000001 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515966+0000", 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3055e-05 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515974+0000", 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3829999999999992e-06 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516216+0000", 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241979 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516223+0000", 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0249999999999997e-06 2024-09-16T03:40:47.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.521421+0000", 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051983740000000004 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.521460+0000", 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8792999999999999e-05 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 53/31 ECSubWrite(tid=19, reqid=client.4615.0:375, at_version=53'12, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.539335+0000", 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.887874346, 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041014900000000002, 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539335+0000", 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539335+0000", 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539335+0000", 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2700000000000001e-07 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539955+0000", 2024-09-16T03:40:47.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00061942200000000001 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539966+0000", 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1048e-05 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539972+0000", 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540035+0000", 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3207999999999995e-05 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540042+0000", 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7190000000000004e-06 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543400+0000", 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033576349999999999 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543437+0000", 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7076999999999998e-05 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 53/31 ECSubWrite(tid=61, reqid=client.4615.0:381, at_version=53'28, trim_to=0'0, roll_forward_to=53'24))", 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.544538+0000", 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.882671271, 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048545120000000001, 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.471 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544538+0000", 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544538+0000", 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544539+0000", 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4900000000000005e-07 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544588+0000", 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9459999999999997e-05 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544600+0000", 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1871e-05 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544608+0000", 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1110000000000008e-06 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544667+0000", 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8557000000000002e-05 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544677+0000", 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0446e-05 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.549354+0000", 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046769180000000004 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.549393+0000", 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8399999999999998e-05 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.475 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 53/31 ECSubWrite(tid=20, reqid=client.4615.0:376, at_version=53'13, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.544800+0000", 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.882409895, 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046257650000000004, 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544800+0000", 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544800+0000", 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544800+0000", 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3799999999999996e-07 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545171+0000", 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037030800000000002 2024-09-16T03:40:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545188+0000", 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6782999999999998e-05 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545197+0000", 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2660000000000007e-06 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545263+0000", 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6612999999999994e-05 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545274+0000", 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.063e-05 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.549399+0000", 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041250449999999999 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.549425+0000", 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6282e-05 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 53/31 ECSubWrite(tid=21, reqid=client.4615.0:377, at_version=53'14, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.545413+0000", 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.881796398000001, 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040345160000000001, 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545413+0000", 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545413+0000", 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545414+0000", 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5199999999999995e-07 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545433+0000", 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8842e-05 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545482+0000", 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9625e-05 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545492+0000", 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3549999999999994e-06 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545988+0000", 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049577900000000001 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.545993+0000", 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5110000000000003e-06 2024-09-16T03:40:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.549428+0000", 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034349929999999999 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.549448+0000", 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9559e-05 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 54/31 ECSubWrite(tid=56, reqid=osd.6.0:56, at_version=54'45, trim_to=0'0, roll_forward_to=53'43))", 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:33.510793+0000", 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.9164165, 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049838840000000001, 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.510793+0000", 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.510793+0000", 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.510793+0000", 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.11e-07 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.510818+0000", 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5194999999999998e-05 2024-09-16T03:40:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511202+0000", 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038325000000000002 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511213+0000", 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1260999999999999e-05 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511276+0000", 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2580000000000001e-05 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511286+0000", 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0029e-05 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.515646+0000", 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043605789999999998 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.515777+0000", 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000130779 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds1 54/31 ECSubWrite(tid=37, reqid=osd.12.0:37, at_version=54'19, trim_to=0'0, roll_forward_to=53'15))", 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:33.510883+0000", 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.91632661, 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004927603, 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.510883+0000", 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.510883+0000", 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.510883+0000", 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7899999999999999e-07 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.510902+0000", 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8821000000000002e-05 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.510910+0000", 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1790000000000008e-06 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.510918+0000", 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3470000000000001e-06 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511243+0000", 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00032578400000000002 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.511252+0000", 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5669999999999995e-06 2024-09-16T03:40:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.515773+0000", 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045209070000000002 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.515811+0000", 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7619e-05 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 54/31 ECSubWrite(tid=57, reqid=osd.6.0:56, at_version=0'0, trim_to=0'0, roll_forward_to=54'45))", 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:33.518441+0000", 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.908768482999999, 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006708416, 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518441+0000", 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518441+0000", 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518442+0000", 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2200000000000003e-07 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518454+0000", 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2285e-05 2024-09-16T03:40:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518465+0000", 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0569e-05 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518470+0000", 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7150000000000003e-06 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518512+0000", 2024-09-16T03:40:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1548000000000003e-05 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518518+0000", 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.331e-06 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525112+0000", 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065939220000000003 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525149+0000", 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7323999999999997e-05 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.0s2 54/31 ECSubWrite(tid=30, reqid=client.4615.0:421, at_version=0'0, trim_to=0'0, roll_forward_to=54'16))", 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:33.518762+0000", 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.908447143, 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063986420000000004, 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518762+0000", 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518762+0000", 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518763+0000", 2024-09-16T03:40:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.13e-07 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518773+0000", 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0448999999999999e-05 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518783+0000", 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5279999999999992e-06 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518788+0000", 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3430000000000003e-06 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518830+0000", 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1984000000000003e-05 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518836+0000", 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.105e-06 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525124+0000", 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062873820000000002 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525161+0000", 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7237999999999998e-05 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 54/31 ECSubWrite(tid=25, reqid=osd.15.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=54'16))", 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:33.521285+0000", 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.905924456999999, 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0042048290000000002, 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521285+0000", 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521285+0000", 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521286+0000", 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8100000000000002e-07 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521299+0000", 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2843000000000001e-05 2024-09-16T03:40:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521324+0000", 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5366000000000001e-05 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521333+0000", 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6400000000000003e-06 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521400+0000", 2024-09-16T03:40:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7176999999999997e-05 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521410+0000", 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0462000000000001e-05 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525457+0000", 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040468910000000004 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525490+0000", 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669000000000002e-05 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 54/31 ECSubWrite(tid=67, reqid=osd.10.0:66, at_version=0'0, trim_to=0'0, roll_forward_to=54'30))", 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:33.521332+0000", 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.905877908000001, 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041759570000000001, 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521332+0000", 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521332+0000", 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521332+0000", 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1699999999999997e-07 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521344+0000", 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2104e-05 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521355+0000", 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0927000000000001e-05 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521363+0000", 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1609999999999997e-06 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521609+0000", 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024603500000000003 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.521618+0000", 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0380000000000005e-06 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525492+0000", 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003874438 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525508+0000", 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5537e-05 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.9s4 63/55 ECSubWrite(tid=64, reqid=osd.1.0:64, at_version=63'41, trim_to=0'0, roll_forward_to=60'39))", 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:42.475171+0000", 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.9520380729999998, 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038144400000000001, 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.475171+0000", 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.475171+0000", 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.475172+0000", 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9000000000000002e-07 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.475192+0000", 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0415e-05 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.475213+0000", 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0789e-05 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.475221+0000", 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6389999999999992e-06 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.475286+0000", 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5037999999999996e-05 2024-09-16T03:40:47.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.475293+0000", 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4669999999999998e-06 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.478952+0000", 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036585889999999999 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.478986+0000", 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4112999999999997e-05 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 63/31 ECSubWrite(tid=31, reqid=osd.15.0:31, at_version=63'24, trim_to=0'0, roll_forward_to=60'22))", 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:42.476967+0000", 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.9502429150000005, 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047634230000000001, 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.516 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476967+0000", 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476967+0000", 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476967+0000", 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477004+0000", 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7030000000000003e-05 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477016+0000", 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2439999999999999e-05 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477025+0000", 2024-09-16T03:40:47.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4810000000000006e-06 2024-09-16T03:40:47.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477112+0000", 2024-09-16T03:40:47.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7298000000000003e-05 2024-09-16T03:40:47.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477122+0000", 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3689999999999996e-06 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.481691+0000", 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045689880000000004 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.481730+0000", 2024-09-16T03:40:47.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9486000000000001e-05 2024-09-16T03:40:47.521 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.521 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.521 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.521 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.521 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.521 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:47.523 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:40:47.524 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T03:40:47.525 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:47.520+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:47.526 INFO:tasks.rados.rados.0.smithi121.stdout:219: done (3 left) 2024-09-16T03:40:47.526 INFO:tasks.rados.rados.0.smithi121.stdout:220: done (2 left) 2024-09-16T03:40:47.526 INFO:tasks.rados.rados.0.smithi121.stdout:221: done (1 left) 2024-09-16T03:40:47.526 INFO:tasks.rados.rados.0.smithi121.stdout:222: done (0 left) 2024-09-16T03:40:47.526 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 29 current snap is 22 2024-09-16T03:40:47.526 INFO:tasks.rados.rados.0.smithi121.stdout:223: seq_num 108 ranges {475136=655360,1130496=606208,1736704=475136,2211840=81920} 2024-09-16T03:40:47.526 INFO:tasks.rados.rados.0.smithi121.stdout:223: writing smithi12138290-29 from 475136 to 1130496 tid 1 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:223: writing smithi12138290-29 from 1130496 to 1736704 tid 2 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:223: writing smithi12138290-29 from 1736704 to 2211840 tid 3 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:223: writing smithi12138290-29 from 2211840 to 2293760 tid 4 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:224: rollback oid 35 current snap is 22 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 35 to 14 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 2 current snap is 22 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:225: seq_num 109 ranges {933888=671744,1605632=753664,2359296=409600} 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:225: writing smithi12138290-2 from 933888 to 1605632 tid 1 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:225: writing smithi12138290-2 from 1605632 to 2359296 tid 2 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:225: writing smithi12138290-2 from 2359296 to 2768896 tid 3 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:226: snap_create 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:223: finishing write tid 1 to smithi12138290-29 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:223: finishing write tid 2 to smithi12138290-29 2024-09-16T03:40:47.527 INFO:tasks.rados.rados.0.smithi121.stdout:224: finishing rollback tid 0 to smithi12138290-35 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:223: finishing write tid 3 to smithi12138290-29 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:224: finishing rollback tid 1 to smithi12138290-35 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:224: finishing rollback tid 2 to smithi12138290-35 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 35 v 66 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:223: finishing write tid 4 to smithi12138290-29 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:223: finishing write tid 5 to smithi12138290-29 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:223: finishing write tid 6 to smithi12138290-29 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:223: oid 29 updating version 0 to 29 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:223: oid 29 updating version 29 to 32 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:223: oid 29 version 32 is already newer than 31 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:223: oid 29 version 32 is already newer than 30 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:223: oid 29 updating version 32 to 33 2024-09-16T03:40:47.528 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 29 v 33 (ObjNum 108 snap 22 seq_num 108) dirty exists 2024-09-16T03:40:47.529 INFO:tasks.rados.rados.0.smithi121.stdout:223: left oid 29 (ObjNum 108 snap 22 seq_num 108) 2024-09-16T03:40:47.529 INFO:tasks.rados.rados.0.smithi121.stdout:225: finishing write tid 1 to smithi12138290-2 2024-09-16T03:40:47.529 INFO:tasks.rados.rados.0.smithi121.stdout:225: finishing write tid 2 to smithi12138290-2 2024-09-16T03:40:47.529 INFO:tasks.rados.rados.0.smithi121.stdout:225: finishing write tid 3 to smithi12138290-2 2024-09-16T03:40:47.529 INFO:tasks.rados.rados.0.smithi121.stdout:225: finishing write tid 4 to smithi12138290-2 2024-09-16T03:40:47.532 INFO:tasks.rados.rados.0.smithi121.stdout:225: finishing write tid 5 to smithi12138290-2 2024-09-16T03:40:47.532 INFO:tasks.rados.rados.0.smithi121.stdout:225: oid 2 updating version 0 to 67 2024-09-16T03:40:47.532 INFO:tasks.rados.rados.0.smithi121.stdout:225: oid 2 version 67 is already newer than 66 2024-09-16T03:40:47.532 INFO:tasks.rados.rados.0.smithi121.stdout:225: oid 2 version 67 is already newer than 65 2024-09-16T03:40:47.533 INFO:tasks.rados.rados.0.smithi121.stdout:225: oid 2 updating version 67 to 68 2024-09-16T03:40:47.533 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 2 v 68 (ObjNum 109 snap 22 seq_num 109) dirty exists 2024-09-16T03:40:47.533 INFO:tasks.rados.rados.0.smithi121.stdout:225: left oid 2 (ObjNum 109 snap 22 seq_num 109) 2024-09-16T03:40:47.623 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:47.621+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:47.668 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:47.668 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:47.668 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:47.668 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:47.677 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:40:47.724 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:47.722+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:47.775 INFO:teuthology.orchestra.run.smithi103.stdout:fsck success 2024-09-16T03:40:47.784 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-09-16T03:40:47.784 INFO:tasks.thrashosds.thrasher:Reviving osd 13 2024-09-16T03:40:47.784 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-16T03:40:47.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-16T03:40:47.820 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:47.820 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:47.820 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:47.820 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:47.820 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:47.825 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:47.823+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:47.827 INFO:tasks.ceph.osd.13:Started 2024-09-16T03:40:47.827 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:40:47.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:40:47.926 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:47.924+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:config diff 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:config diff get 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:config get 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:config help [] 2024-09-16T03:40:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:config set ... 2024-09-16T03:40:47.928 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: invalid command 2024-09-16T03:40:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:40:47.929 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-13, ['dump_ops_in_flight'] 2024-09-16T03:40:47.935 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:47.935 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:47.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:47.935 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4341.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:39:59.266198+0000", 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 48.666062265000001, 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035726070000000002, 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4341", 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5 2024-09-16T03:40:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266198+0000", 2024-09-16T03:40:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266198+0000", 2024-09-16T03:40:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266200+0000", 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5620000000000002e-06 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266214+0000", 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3709999999999999e-05 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266236+0000", 2024-09-16T03:40:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2407000000000001e-05 2024-09-16T03:40:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266247+0000", 2024-09-16T03:40:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0152999999999999e-05 2024-09-16T03:40:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266319+0000", 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2133999999999997e-05 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.266328+0000", 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0799999999999995e-06 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.268192+0000", 2024-09-16T03:40:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018645529999999999 2024-09-16T03:40:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.268308+0000", 2024-09-16T03:40:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011581100000000001 2024-09-16T03:40:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 10", 2024-09-16T03:40:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.268416+0000", 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010828300000000001 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269196+0000", 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077989699999999999 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269705+0000", 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00050892999999999999 2024-09-16T03:40:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269742+0000", 2024-09-16T03:40:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7128000000000001e-05 2024-09-16T03:40:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:39:59.269770+0000", 2024-09-16T03:40:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7959000000000001e-05 2024-09-16T03:40:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/31 ECSubWrite(tid=23, reqid=client.4615.0:205, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.407069974999999, 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035619599999999999, 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525190+0000", 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.31e-07 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525645+0000", 2024-09-16T03:40:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045454399999999998 2024-09-16T03:40:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525658+0000", 2024-09-16T03:40:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3159999999999999e-05 2024-09-16T03:40:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525669+0000", 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1134999999999999e-05 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525736+0000", 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7175000000000007e-05 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.525746+0000", 2024-09-16T03:40:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.927e-06 2024-09-16T03:40:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528720+0000", 2024-09-16T03:40:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029734119999999999 2024-09-16T03:40:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:16.528752+0000", 2024-09-16T03:40:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2475999999999999e-05 2024-09-16T03:40:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:229 3.5s0 3:a4bec39a:::smithi12138290-4:head [copy-from ver 9] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:40:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.391683816, 2024-09-16T03:40:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016562443, 2024-09-16T03:40:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 229 2024-09-16T03:40:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:40:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540579+0000", 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6910000000000002e-06 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540591+0000", 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2018000000000001e-05 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540631+0000", 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9938999999999998e-05 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540643+0000", 2024-09-16T03:40:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1902000000000001e-05 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540731+0000", 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8585000000000007e-05 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540736+0000", 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9919999999999998e-06 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542533+0000", 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017970950000000001 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542727+0000", 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019304700000000001 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.553519+0000", 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010792568000000001 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554988+0000", 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014687490000000001 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557059+0000", 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00207092 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557139+0000", 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9937000000000004e-05 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:230 3.5s0 3:a4bec39a:::smithi12138290-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.391572710999998, 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014084419000000001, 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 230 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540688+0000", 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540692+0000", 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.019e-06 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540724+0000", 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2237000000000003e-05 2024-09-16T03:40:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540740+0000", 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6075000000000001e-05 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540783+0000", 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2249000000000002e-05 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540786+0000", 2024-09-16T03:40:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4869999999999998e-06 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543015+0000", 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022290859999999999 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543063+0000", 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7496000000000001e-05 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554628+0000", 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011565203 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554697+0000", 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9381999999999994e-05 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554772+0000", 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4535000000000004e-05 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 42/31 ECSubWrite(tid=30, reqid=osd.14.0:30, at_version=42'20, trim_to=0'0, roll_forward_to=41'18))", 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.374685412000002, 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048714179999999998, 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557575+0000", 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3799999999999999e-07 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557594+0000", 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9415e-05 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557606+0000", 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1354e-05 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557615+0000", 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9600000000000006e-06 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557669+0000", 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4024000000000003e-05 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.557674+0000", 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.587e-06 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.562413+0000", 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004739254 2024-09-16T03:40:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.562446+0000", 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2586000000000003e-05 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:314 3.5s0 3:a4bec39a:::smithi12138290-4:head [stat,delete] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.441506718999999, 2024-09-16T03:40:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010497464, 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 314 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:40:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490755+0000", 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.736e-06 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490767+0000", 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1931e-05 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490783+0000", 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6135000000000001e-05 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490791+0000", 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1620000000000008e-06 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490849+0000", 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7581000000000003e-05 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490856+0000", 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9299999999999997e-06 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.492657+0000", 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018011539999999999 2024-09-16T03:40:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.492731+0000", 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4263000000000004e-05 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.493469+0000", 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073743400000000003 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.495358+0000", 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001888898 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.501195+0000", 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058370879999999998 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.501251+0000", 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6152e-05 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 53/31 ECSubWrite(tid=52, reqid=osd.14.0:52, at_version=53'41, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.502471+0000", 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.429788693000001, 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041823930000000004, 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502471+0000", 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502471+0000", 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502472+0000", 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.034e-06 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502498+0000", 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5514999999999999e-05 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502510+0000", 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2465e-05 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502519+0000", 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7460000000000006e-06 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502597+0000", 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8149e-05 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502606+0000", 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0880000000000001e-06 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506601+0000", 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039948939999999997 2024-09-16T03:40:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506654+0000", 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2502e-05 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 53/31 ECSubWrite(tid=53, reqid=osd.14.0:53, at_version=53'43, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.502542+0000", 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.429718141, 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049945160000000001, 2024-09-16T03:40:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502542+0000", 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502542+0000", 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502542+0000", 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502570+0000", 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7962000000000001e-05 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502578+0000", 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0739999999999993e-06 2024-09-16T03:40:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502585+0000", 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6089999999999997e-06 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502929+0000", 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034402599999999999 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502938+0000", 2024-09-16T03:40:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0289999999999994e-06 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507478+0000", 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045398050000000001 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507536+0000", 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8610999999999998e-05 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=27, reqid=client.4615.0:354, at_version=53'8, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.427309190000001, 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055158589999999997, 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:40:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:40:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1e-07 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504976+0000", 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5259999999999999e-05 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504984+0000", 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6850000000000001e-06 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504993+0000", 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4299999999999995e-06 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505310+0000", 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031660299999999999 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505342+0000", 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.15e-05 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510423+0000", 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050819000000000003 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510467+0000", 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3271e-05 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.505340+0000", 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.426920083000001, 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054441530000000002, 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505340+0000", 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505340+0000", 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505341+0000", 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505367+0000", 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6414000000000002e-05 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505380+0000", 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2599000000000001e-05 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505388+0000", 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2900000000000002e-06 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507195+0000", 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001806698 2024-09-16T03:40:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507205+0000", 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0057999999999999e-05 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510749+0000", 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035439759999999999 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510784+0000", 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5351000000000001e-05 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=29, reqid=client.4615.0:356, at_version=53'10, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.505935+0000", 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.42632457, 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048734399999999997, 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505935+0000", 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505935+0000", 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505936+0000", 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1999999999999999e-07 2024-09-16T03:40:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505955+0000", 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8726999999999999e-05 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505970+0000", 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5254e-05 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505981+0000", 2024-09-16T03:40:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0421000000000001e-05 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507941+0000", 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019600279999999999 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507952+0000", 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1524e-05 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510788+0000", 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002835522 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510809+0000", 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1243999999999999e-05 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=30, reqid=client.4615.0:357, at_version=53'12, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.508937+0000", 2024-09-16T03:40:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.423322679, 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037675410000000001, 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508937+0000", 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508937+0000", 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508938+0000", 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1999999999999998e-07 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509070+0000", 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013158500000000001 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509081+0000", 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1303e-05 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509089+0000", 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7510000000000008e-06 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509138+0000", 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9014000000000002e-05 2024-09-16T03:40:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509146+0000", 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2330000000000006e-06 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512665+0000", 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00351914 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512705+0000", 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9595000000000003e-05 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=31, reqid=client.4615.0:358, at_version=53'13, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.511311+0000", 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.420948783, 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004370306, 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511311+0000", 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511311+0000", 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511312+0000", 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8800000000000002e-07 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511418+0000", 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010646599999999999 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511430+0000", 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1222e-05 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511439+0000", 2024-09-16T03:40:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9679999999999995e-06 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511584+0000", 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000145315 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511596+0000", 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2329e-05 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515644+0000", 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00404762 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515682+0000", 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7697999999999997e-05 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=32, reqid=client.4615.0:359, at_version=53'14, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.513931+0000", 2024-09-16T03:40:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.418329394000001, 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004942264, 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513931+0000", 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513931+0000", 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513932+0000", 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514010+0000", 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8285e-05 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514023+0000", 2024-09-16T03:40:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2666999999999999e-05 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514032+0000", 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3999999999999998e-06 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514382+0000", 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000349742 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.514391+0000", 2024-09-16T03:40:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.986e-06 2024-09-16T03:40:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518837+0000", 2024-09-16T03:40:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044466339999999997 2024-09-16T03:40:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518873+0000", 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5605999999999998e-05 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s1 53/31 ECSubWrite(tid=19, reqid=client.4615.0:375, at_version=53'12, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.539243+0000", 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.393017051999999, 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043355520000000003, 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539243+0000", 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539243+0000", 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539248+0000", 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8420000000000004e-06 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539870+0000", 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062189099999999996 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539884+0000", 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4644000000000001e-05 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539894+0000", 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2390000000000008e-06 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539963+0000", 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9496e-05 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539974+0000", 2024-09-16T03:40:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0592e-05 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543541+0000", 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035675329999999999 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543579+0000", 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7314999999999998e-05 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:467 3.5s0 3:a45e0081:::smithi12138290-25:head [copy-from ver 16] snapc 11=[11,10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.449428975, 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014721097000000001, 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 467 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482832+0000", 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3e-06 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482839+0000", 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3010000000000003e-06 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482849+0000", 2024-09-16T03:40:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9499999999999996e-06 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482854+0000", 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1410000000000004e-06 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482916+0000", 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2040999999999998e-05 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483028+0000", 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112674 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.491088+0000", 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080595969999999999 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.493079+0000", 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019907990000000001 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.497458+0000", 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004378804 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.497552+0000", 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4489999999999998e-05 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:468 3.5s0 3:a45e0081:::smithi12138290-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:40:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.4493769959999998, 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010159837999999999, 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 468 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482884+0000", 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.16e-07 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482888+0000", 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0099999999999997e-06 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482903+0000", 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5418999999999999e-05 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482910+0000", 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6850000000000004e-06 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483315+0000", 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040511699999999998 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483369+0000", 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3884000000000001e-05 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492657+0000", 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092879590000000001 2024-09-16T03:40:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492897+0000", 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023993300000000001 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.493043+0000", 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014601499999999999 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s1 63/31 ECSubWrite(tid=31, reqid=osd.15.0:31, at_version=63'24, trim_to=0'0, roll_forward_to=60'22))", 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:42.476343+0000", 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.4559166110000001, 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049178959999999997, 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476343+0000", 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476343+0000", 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476344+0000", 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476362+0000", 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8059000000000001e-05 2024-09-16T03:40:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476374+0000", 2024-09-16T03:40:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2323e-05 2024-09-16T03:40:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476382+0000", 2024-09-16T03:40:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9920000000000007e-06 2024-09-16T03:40:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476480+0000", 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8263000000000004e-05 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.476489+0000", 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.687e-06 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.481234+0000", 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047449370000000003 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.481261+0000", 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7305e-05 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 63/31 ECSubWrite(tid=79, reqid=osd.14.0:78, at_version=0'0, trim_to=0'0, roll_forward_to=63'66))", 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:42.477374+0000", 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.4548860570000004, 2024-09-16T03:40:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039441600000000004, 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477374+0000", 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477374+0000", 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477375+0000", 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8199999999999998e-07 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477388+0000", 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3385e-05 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477399+0000", 2024-09-16T03:40:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1018e-05 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477407+0000", 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8699999999999992e-06 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477480+0000", 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2997000000000006e-05 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.477489+0000", 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1849999999999993e-06 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.481279+0000", 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037894769999999999 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:42.481318+0000", 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9345999999999999e-05 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 68/31 ECSubWrite(tid=80, reqid=client.4615.0:578, at_version=68'29, trim_to=0'0, roll_forward_to=63'27))", 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:47.485308+0000", 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.44695168800000001, 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039904559999999999, 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485308+0000", 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485308+0000", 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485309+0000", 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9000000000000002e-07 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485428+0000", 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011880200000000001 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485442+0000", 2024-09-16T03:40:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4542000000000001e-05 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485451+0000", 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2669999999999995e-06 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485523+0000", 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1852000000000003e-05 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485542+0000", 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8355999999999998e-05 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489264+0000", 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037223069999999998 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489299+0000", 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4940000000000001e-05 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.041 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.041 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.041 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:48.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:40:48.043 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:48.041+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:48.143 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:48.141+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:48.149 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:48.149 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:48.156 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:40:48.244 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:48.242+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:48.259 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:48.259 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:48.259 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:48.259 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:48.259 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:48.267 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:40:48.345 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:48.343+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:48.373 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:48.373 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:48.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:48.373 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:48.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:253 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.791387960000002, 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084206999999999997, 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 253 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579061+0000", 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3350000000000001e-06 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579072+0000", 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0964999999999999e-05 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579093+0000", 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1766000000000001e-05 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579104+0000", 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0271999999999999e-05 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579200+0000", 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5917999999999999e-05 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579208+0000", 2024-09-16T03:40:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9479999999999997e-06 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581270+0000", 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020611140000000002 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581437+0000", 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016734000000000001 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587479+0000", 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060420420000000001 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:254 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.791304874000001, 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087869469999999998, 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 254 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579142+0000", 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579146+0000", 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0600000000000001e-06 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579155+0000", 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7620000000000001e-06 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579161+0000", 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8640000000000001e-06 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579289+0000", 2024-09-16T03:40:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000128677 2024-09-16T03:40:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579298+0000", 2024-09-16T03:40:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4100000000000008e-06 2024-09-16T03:40:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581916+0000", 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026177570000000001 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581995+0000", 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9833999999999995e-05 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587928+0000", 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059329329999999996 2024-09-16T03:40:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:255 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.791260956999999, 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090005769999999992, 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 255 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579186+0000", 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9500000000000002e-07 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579189+0000", 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579197+0000", 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8189999999999992e-06 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579202+0000", 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579337+0000", 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134547 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579343+0000", 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583029+0000", 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036861540000000001 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583113+0000", 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3370999999999998e-05 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.588186+0000", 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050730269999999999 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:354 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.498416+0000", 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.872030478999999, 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017750095, 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 354 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498416+0000", 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498416+0000", 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498418+0000", 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0899999999999999e-06 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498426+0000", 2024-09-16T03:40:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.909e-06 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498442+0000", 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6164999999999999e-05 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498451+0000", 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5240000000000006e-06 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498523+0000", 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1482000000000004e-05 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498535+0000", 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2072000000000001e-05 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502229+0000", 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003693806 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502333+0000", 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104041 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503454+0000", 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011210440000000001 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515617+0000", 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012162681999999999 2024-09-16T03:40:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516067+0000", 2024-09-16T03:40:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000450817 2024-09-16T03:40:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516166+0000", 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8462999999999996e-05 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:355 3.ds0 3:b952e66c:::smithi12138290-29:head [rollback b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.871971607000001, 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017895208999999999, 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 355 2024-09-16T03:40:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:40:48.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:40:48.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8100000000000003e-07 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498478+0000", 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8049999999999998e-06 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498485+0000", 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0219999999999999e-06 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498489+0000", 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9829999999999998e-06 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498604+0000", 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011544699999999999 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498609+0000", 2024-09-16T03:40:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2760000000000002e-06 2024-09-16T03:40:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503989+0000", 2024-09-16T03:40:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053800360000000004 2024-09-16T03:40:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504091+0000", 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010255300000000001 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504699+0000", 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00060748699999999998 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515811+0000", 2024-09-16T03:40:48.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011112662000000001 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516302+0000", 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049017500000000003 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516370+0000", 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8281999999999994e-05 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.404 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:356 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.498505+0000", 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.871940958, 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018089424, 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 356 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498505+0000", 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498505+0000", 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498506+0000", 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3700000000000001e-07 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498508+0000", 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498513+0000", 2024-09-16T03:40:48.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4519999999999996e-06 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498518+0000", 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6399999999999996e-06 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498643+0000", 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125265 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498648+0000", 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0320000000000003e-06 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505217+0000", 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065689839999999999 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505317+0000", 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9613999999999999e-05 2024-09-16T03:40:48.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505895+0000", 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057821100000000002 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515868+0000", 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099727659999999992 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516509+0000", 2024-09-16T03:40:48.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00064155600000000005 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516595+0000", 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5206999999999999e-05 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:357 3.ds0 3:b84aed6c:::smithi12138290-13:head [append 0~622592] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.504582+0000", 2024-09-16T03:40:48.411 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.865864386, 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012468976999999999, 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 357 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504582+0000", 2024-09-16T03:40:48.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504582+0000", 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504584+0000", 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.057e-06 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505874+0000", 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012901060000000001 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505888+0000", 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3689999999999999e-05 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505896+0000", 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3580000000000003e-06 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506261+0000", 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000365239 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506357+0000", 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5466999999999999e-05 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508262+0000", 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001905622 2024-09-16T03:40:48.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516183+0000", 2024-09-16T03:40:48.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079204419999999998 2024-09-16T03:40:48.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516969+0000", 2024-09-16T03:40:48.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00078583600000000004 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517051+0000", 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2159999999999999e-05 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:358 3.ds0 3:b84aed6c:::smithi12138290-13:head [append 0~475136] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.507540+0000", 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.862906599, 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011581542, 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 358 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507540+0000", 2024-09-16T03:40:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507540+0000", 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507541+0000", 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.716e-06 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507949+0000", 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040747899999999998 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507963+0000", 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3828999999999999e-05 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507971+0000", 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7430000000000008e-06 2024-09-16T03:40:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510078+0000", 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002106293 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510200+0000", 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012225100000000001 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511236+0000", 2024-09-16T03:40:48.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010360289999999999 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516185+0000", 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049487079999999996 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519053+0000", 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028678610000000002 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519121+0000", 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8633000000000005e-05 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:359 3.ds0 3:b84aed6c:::smithi12138290-13:head [append 0~294912] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.509631+0000", 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.860814786000001, 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097547659999999998, 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.423 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 359 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509631+0000", 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509631+0000", 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509634+0000", 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1459999999999999e-06 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509870+0000", 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023675700000000001 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509882+0000", 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1905999999999999e-05 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509891+0000", 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9870000000000005e-06 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512302+0000", 2024-09-16T03:40:48.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002410812 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512443+0000", 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014075700000000001 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513286+0000", 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000843021 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516670+0000", 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033843889999999998 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519344+0000", 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026737129999999999 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519386+0000", 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2277999999999997e-05 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.428 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:360 3.ds0 3:b84aed6c:::smithi12138290-13:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.860537569, 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010089805, 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.429 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 360 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8999999999999996e-07 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509913+0000", 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0269999999999997e-06 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509922+0000", 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.687e-06 2024-09-16T03:40:48.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509928+0000", 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8000000000000004e-06 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517293+0000", 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073646559999999998 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517402+0000", 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109687 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517900+0000", 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049748700000000002 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519217+0000", 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001317033 2024-09-16T03:40:48.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519941+0000", 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00072411000000000003 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519998+0000", 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7627999999999998e-05 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:361 3.ds0 3:b84aed6c:::smithi12138290-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.509950+0000", 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.860495752, 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01277462, 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 361 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.435 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509950+0000", 2024-09-16T03:40:48.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509950+0000", 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509951+0000", 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7299999999999996e-07 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509956+0000", 2024-09-16T03:40:48.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4739999999999999e-06 2024-09-16T03:40:48.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509963+0000", 2024-09-16T03:40:48.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:40:48.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509968+0000", 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4530000000000001e-06 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518702+0000", 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087336250000000001 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518806+0000", 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104905 2024-09-16T03:40:48.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.520329+0000", 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015226879999999999 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.520422+0000", 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3224000000000001e-05 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.522725+0000", 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023026980000000002 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s6 53/31 ECSubWrite(tid=19, reqid=client.4615.0:375, at_version=53'12, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.539609+0000", 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.830837172000001, 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039602220000000002, 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.441 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539609+0000", 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539609+0000", 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.539609+0000", 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.49e-07 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540054+0000", 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000444988 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540068+0000", 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3388e-05 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540077+0000", 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1349999999999998e-06 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540156+0000", 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9769000000000001e-05 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.540166+0000", 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8090000000000004e-06 2024-09-16T03:40:48.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543538+0000", 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033718910000000001 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543569+0000", 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1093000000000002e-05 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s6 53/31 ECSubWrite(tid=22, reqid=client.4615.0:377, at_version=0'0, trim_to=0'0, roll_forward_to=53'14))", 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.551278+0000", 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.819168678, 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034132730000000001, 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551278+0000", 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551278+0000", 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551279+0000", 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5290000000000001e-06 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551293+0000", 2024-09-16T03:40:48.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.415e-05 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551316+0000", 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2552e-05 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551327+0000", 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0858e-05 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551400+0000", 2024-09-16T03:40:48.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3107000000000003e-05 2024-09-16T03:40:48.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551409+0000", 2024-09-16T03:40:48.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3840000000000003e-06 2024-09-16T03:40:48.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554640+0000", 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032307239999999999 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554691+0000", 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0968999999999999e-05 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.cs2 53/31 ECSubWrite(tid=92, reqid=client.4615.0:388, at_version=53'27, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T03:40:48.450 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.552782+0000", 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.817664189, 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037395129999999999, 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552782+0000", 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552782+0000", 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552783+0000", 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1000000000000001e-06 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552866+0000", 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2631000000000002e-05 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552887+0000", 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1715000000000001e-05 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552898+0000", 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.08e-05 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552979+0000", 2024-09-16T03:40:48.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1166000000000002e-05 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552990+0000", 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0679000000000001e-05 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.556491+0000", 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035008600000000002 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.556522+0000", 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0562000000000003e-05 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:578 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~655360] snapc 17=[17,16,15,13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:47.482693+0000", 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.88775345900000002, 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093753210000000007, 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.455 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 578 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.482693+0000", 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.482693+0000", 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.482695+0000", 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5000000000000002e-06 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.483400+0000", 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00070434899999999995 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.483422+0000", 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1963999999999998e-05 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.483434+0000", 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2867000000000001e-05 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.483544+0000", 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109376 2024-09-16T03:40:48.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.483621+0000", 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6940000000000005e-05 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.484484+0000", 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00086355600000000002 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.488177+0000", 2024-09-16T03:40:48.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00369251 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492014+0000", 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038367269999999998 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492068+0000", 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4531999999999999e-05 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.460 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:579 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~606208] snapc 17=[17,16,15,13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:47.485904+0000", 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.88454242199999999, 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063398630000000003, 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.461 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 579 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485904+0000", 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485904+0000", 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.485905+0000", 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5319999999999999e-06 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.486502+0000", 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00059661299999999996 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.486515+0000", 2024-09-16T03:40:48.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3416e-05 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.486523+0000", 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3749999999999997e-06 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.486584+0000", 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0807999999999998e-05 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.486651+0000", 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7533000000000005e-05 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.487284+0000", 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00063241299999999996 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.488745+0000", 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001461579 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492198+0000", 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034527519999999999 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492244+0000", 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5841999999999999e-05 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:580 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~475136] snapc 17=[17,16,15,13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T03:40:48.466 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:47.488420+0000", 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.88202623599999996, 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058408729999999999, 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 580 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.488420+0000", 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.488420+0000", 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.488421+0000", 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4279999999999999e-06 2024-09-16T03:40:48.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489096+0000", 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067451499999999999 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489110+0000", 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4541e-05 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489119+0000", 2024-09-16T03:40:48.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6309999999999992e-06 2024-09-16T03:40:48.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489175+0000", 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5693999999999999e-05 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489267+0000", 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2570000000000003e-05 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489955+0000", 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00068775599999999998 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492317+0000", 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023619829999999998 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.494200+0000", 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018830380000000001 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.494261+0000", 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0717000000000002e-05 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:581 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~81920] snapc 17=[17,16,15,13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:47.489137+0000", 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.881309067, 2024-09-16T03:40:48.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058213960000000004, 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 581 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489137+0000", 2024-09-16T03:40:48.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489137+0000", 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489138+0000", 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0020000000000001e-06 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489204+0000", 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5976000000000007e-05 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489214+0000", 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0033e-05 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489221+0000", 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5719999999999999e-06 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.490516+0000", 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012952110000000001 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.490895+0000", 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037924299999999997 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.491454+0000", 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00055925400000000002 2024-09-16T03:40:48.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.493608+0000", 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021535769999999998 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.494921+0000", 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013131869999999999 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.494959+0000", 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7341e-05 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:582 3.ds0 3:b952e66c:::smithi12138290-29:head [setxattr _header (57) in=7b] snapc 17=[17,16,15,13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:47.489234+0000", 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.88121260800000001, 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059141840000000003, 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.479 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 582 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489234+0000", 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489234+0000", 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489234+0000", 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7e-07 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489237+0000", 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4380000000000002e-06 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489241+0000", 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3649999999999997e-06 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489244+0000", 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3840000000000001e-06 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492419+0000", 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031745229999999998 2024-09-16T03:40:48.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492475+0000", 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6484999999999999e-05 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492771+0000", 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029534199999999998 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.494271+0000", 2024-09-16T03:40:48.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015005680000000001 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.495067+0000", 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00079570699999999999 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.495148+0000", 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0902000000000006e-05 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.484 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:583 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:47.489264+0000", 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.88118257700000002, 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072002020000000002, 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 583 2024-09-16T03:40:48.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489264+0000", 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489264+0000", 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489264+0000", 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7999999999999995e-07 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489267+0000", 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5270000000000001e-06 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489281+0000", 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3743e-05 2024-09-16T03:40:48.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.489287+0000", 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0850000000000002e-06 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492998+0000", 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037113950000000001 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.493085+0000", 2024-09-16T03:40:48.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6873999999999995e-05 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.495521+0000", 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002436602 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.495614+0000", 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2614999999999993e-05 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:48.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:48.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.496464+0000", 2024-09-16T03:40:48.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084978099999999997 2024-09-16T03:40:48.490 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.490 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.490 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.490 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:48.490 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:48.490 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:48.492 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:40:48.492 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:48.489+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:48.498 INFO:tasks.rados.rados.0.smithi121.stdout:223: done (3 left) 2024-09-16T03:40:48.499 INFO:tasks.rados.rados.0.smithi121.stdout:224: done (2 left) 2024-09-16T03:40:48.499 INFO:tasks.rados.rados.0.smithi121.stdout:225: done (1 left) 2024-09-16T03:40:48.499 INFO:tasks.rados.rados.0.smithi121.stdout:226: done (0 left) 2024-09-16T03:40:48.499 INFO:tasks.rados.rados.0.smithi121.stdout:227: delete oid 1 current snap is 23 2024-09-16T03:40:48.511 INFO:tasks.rados.rados.0.smithi121.stdout:227: done (0 left) 2024-09-16T03:40:48.511 INFO:tasks.rados.rados.0.smithi121.stdout:228: read oid 35 snap -1 2024-09-16T03:40:48.511 INFO:tasks.rados.rados.0.smithi121.stdout:228: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:40:48.511 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 41 current snap is 23 2024-09-16T03:40:48.511 INFO:tasks.rados.rados.0.smithi121.stdout:229: seq_num 111 ranges {688128=704512,1392640=655360,2048000=507904} 2024-09-16T03:40:48.519 INFO:tasks.rados.rados.0.smithi121.stdout:229: writing smithi12138290-41 from 688128 to 1392640 tid 1 2024-09-16T03:40:48.522 INFO:tasks.rados.rados.0.smithi121.stdout:229: writing smithi12138290-41 from 1392640 to 2048000 tid 2 2024-09-16T03:40:48.525 INFO:tasks.rados.rados.0.smithi121.stdout:229: writing smithi12138290-41 from 2048000 to 2555904 tid 3 2024-09-16T03:40:48.525 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 36 current snap is 23 2024-09-16T03:40:48.525 INFO:tasks.rados.rados.0.smithi121.stdout:230: seq_num 112 ranges {0=720896,720896=802816,1523712=360448} 2024-09-16T03:40:48.529 INFO:tasks.rados.rados.0.smithi121.stdout:230: writing smithi12138290-36 from 0 to 720896 tid 1 2024-09-16T03:40:48.534 INFO:tasks.rados.rados.0.smithi121.stdout:230: writing smithi12138290-36 from 720896 to 1523712 tid 2 2024-09-16T03:40:48.536 INFO:tasks.rados.rados.0.smithi121.stdout:230: writing smithi12138290-36 from 1523712 to 1884160 tid 3 2024-09-16T03:40:48.536 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 13 current snap is 23 2024-09-16T03:40:48.536 INFO:tasks.rados.rados.0.smithi121.stdout:231: seq_num 113 ranges {0=737280,737280=557056,1294336=606208} 2024-09-16T03:40:48.541 INFO:tasks.rados.rados.0.smithi121.stdout:231: writing smithi12138290-13 from 0 to 737280 tid 1 2024-09-16T03:40:48.544 INFO:tasks.rados.rados.0.smithi121.stdout:231: writing smithi12138290-13 from 737280 to 1294336 tid 2 2024-09-16T03:40:48.547 INFO:tasks.rados.rados.0.smithi121.stdout:231: writing smithi12138290-13 from 1294336 to 1900544 tid 3 2024-09-16T03:40:48.547 INFO:tasks.rados.rados.0.smithi121.stdout:232: snap_remove snap 21 2024-09-16T03:40:48.592 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:48.591+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:48.602 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:48.602 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:48.611 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:40:48.693 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:48.690+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:48.714 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:48.714 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:40:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:40:48.715 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:48.724 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:40:48.808 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:48.806+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:48.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:48.868 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs5 38/31 ECSubWrite(tid=63, reqid=client.4615.0:226, at_version=38'10, trim_to=0'0, roll_forward_to=34'7))", 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.544170+0000", 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.321939712999999, 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056030519999999999, 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544170+0000", 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544170+0000", 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544171+0000", 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4300000000000003e-07 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544310+0000", 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013949599999999999 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544330+0000", 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9539e-05 2024-09-16T03:40:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544342+0000", 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.155e-05 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544395+0000", 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3318e-05 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.544404+0000", 2024-09-16T03:40:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.149e-06 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549741+0000", 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005337456 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549773+0000", 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1900999999999998e-05 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:251 3.0s0 3:00cc7e13:::smithi12138290-42:head [copy-from ver 4] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.309374213000002, 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013120588000000001, 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 251 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556738+0000", 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8989999999999999e-06 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556745+0000", 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556759+0000", 2024-09-16T03:40:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.451e-05 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556766+0000", 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1260000000000003e-06 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556817+0000", 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1693999999999997e-05 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556824+0000", 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8549999999999996e-06 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557940+0000", 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001115888 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558009+0000", 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8707999999999993e-05 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564426+0000", 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064170659999999999 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.566682+0000", 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022565039999999999 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569779+0000", 2024-09-16T03:40:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030962770000000001 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569856+0000", 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7843999999999995e-05 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:252 3.0s0 3:00cc7e13:::smithi12138290-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.556781+0000", 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.309329082000001, 2024-09-16T03:40:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008452701, 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 252 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556781+0000", 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556781+0000", 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556782+0000", 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8899999999999999e-07 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556786+0000", 2024-09-16T03:40:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0910000000000003e-06 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556795+0000", 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1249999999999999e-06 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556801+0000", 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1040000000000003e-06 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556876+0000", 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5113999999999999e-05 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556880+0000", 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2749999999999997e-06 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558172+0000", 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001291936 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558261+0000", 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8628999999999995e-05 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.565110+0000", 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068490360000000002 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.565168+0000", 2024-09-16T03:40:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8087999999999997e-05 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.565234+0000", 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5714000000000001e-05 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:332 3.0s0 3:0cbe5f28:::smithi12138290-39:head [append 0~589824] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.475060+0000", 2024-09-16T03:40:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.391050302, 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0090762959999999993, 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 332 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475060+0000", 2024-09-16T03:40:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475060+0000", 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475062+0000", 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.644e-06 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475787+0000", 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00072438999999999999 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475800+0000", 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3288000000000001e-05 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475807+0000", 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1779999999999997e-06 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475858+0000", 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.062e-05 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.475864+0000", 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3589999999999996e-06 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.477601+0000", 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001736455 2024-09-16T03:40:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.477664+0000", 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2928999999999993e-05 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478538+0000", 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00087489000000000002 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.480310+0000", 2024-09-16T03:40:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001771044 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.484058+0000", 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037481099999999998 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.484136+0000", 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8388999999999998e-05 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:333 3.0s0 3:0cbe5f28:::smithi12138290-39:head [append 0~573440] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.478105+0000", 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.388004828, 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006433935, 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 333 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478105+0000", 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478105+0000", 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478106+0000", 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1200000000000001e-06 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478666+0000", 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00055984100000000005 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478679+0000", 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3067999999999999e-05 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.478688+0000", 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0180000000000007e-06 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479162+0000", 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000474177 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479259+0000", 2024-09-16T03:40:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6261999999999994e-05 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479878+0000", 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061973200000000001 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.482152+0000", 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022740899999999999 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.484493+0000", 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023406490000000002 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.484539+0000", 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5977999999999999e-05 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:334 3.0s0 3:0cbe5f28:::smithi12138290-39:head [append 0~196608] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.479263+0000", 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.386846524999999, 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093778520000000008, 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 334 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479263+0000", 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479263+0000", 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479264+0000", 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9999999999999995e-07 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479450+0000", 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018548100000000001 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479460+0000", 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8030000000000008e-06 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479465+0000", 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9049999999999996e-06 2024-09-16T03:40:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.480432+0000", 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00096703299999999996 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.480512+0000", 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9793e-05 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.480999+0000", 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048716200000000002 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.488494+0000", 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074957039999999997 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.488566+0000", 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1384000000000006e-05 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.488641+0000", 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5587000000000007e-05 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:335 3.0s0 3:0cbe5f28:::smithi12138290-39:head [setxattr _header (57) in=7b] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.479477+0000", 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.386633136, 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096011870000000006, 2024-09-16T03:40:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 335 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479477+0000", 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479477+0000", 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479477+0000", 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7100000000000002e-07 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479480+0000", 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.711e-06 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479486+0000", 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5879999999999997e-06 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479489+0000", 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2399999999999999e-06 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481425+0000", 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019356620000000001 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481515+0000", 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0655000000000005e-05 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.481869+0000", 2024-09-16T03:40:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00035413799999999999 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.488659+0000", 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067899839999999998 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.488716+0000", 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6205999999999997e-05 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.489078+0000", 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036243199999999999 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:336 3.0s0 3:0cbe5f28:::smithi12138290-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.479494+0000", 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.386615801000001, 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01140001, 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 336 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479494+0000", 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479494+0000", 2024-09-16T03:40:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479495+0000", 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4900000000000001e-07 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479497+0000", 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1260000000000001e-06 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479503+0000", 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479506+0000", 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.482263+0000", 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027574219999999998 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.482312+0000", 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9196999999999997e-05 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.489119+0000", 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068063610000000004 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.489197+0000", 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8570000000000002e-05 2024-09-16T03:40:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.490894+0000", 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001697034 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 53/31 ECSubWrite(tid=50, reqid=client.4615.0:349, at_version=53'38, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.500322+0000", 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.365788308999999, 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0061250009999999997, 2024-09-16T03:40:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500322+0000", 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500322+0000", 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500323+0000", 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.153e-06 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500345+0000", 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2157000000000001e-05 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500358+0000", 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2986999999999999e-05 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500367+0000", 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1500000000000005e-06 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.501187+0000", 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081999099999999995 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.501198+0000", 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.046e-05 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.506404+0000", 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052068180000000002 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.506447+0000", 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2284999999999999e-05 2024-09-16T03:40:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 53/31 ECSubWrite(tid=51, reqid=client.4615.0:350, at_version=53'39, trim_to=0'0, roll_forward_to=51'35))", 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.500839+0000", 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.365271101000001, 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056341550000000001, 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500839+0000", 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500839+0000", 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500839+0000", 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8299999999999997e-07 2024-09-16T03:40:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500859+0000", 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9170000000000001e-05 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500873+0000", 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3893000000000001e-05 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.500881+0000", 2024-09-16T03:40:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4659999999999999e-06 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.501722+0000", 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00084063499999999995 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.501731+0000", 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6080000000000002e-06 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.506451+0000", 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047198630000000004 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.506473+0000", 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1937e-05 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.3s4 53/31 ECSubWrite(tid=19, reqid=client.4615.0:375, at_version=53'12, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:32.539176+0000", 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.326934324, 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052376710000000002, 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539176+0000", 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539176+0000", 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539176+0000", 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9499999999999998e-07 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539681+0000", 2024-09-16T03:40:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050504399999999996 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539693+0000", 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1770000000000001e-05 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539701+0000", 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5609999999999999e-06 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539762+0000", 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0242999999999999e-05 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.539772+0000", 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0129000000000001e-05 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.544371+0000", 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045991950000000004 2024-09-16T03:40:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:32.544413+0000", 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2333999999999998e-05 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:421 3.0s0 3:0cbe5f28:::smithi12138290-39:head [stat,delete] snapc d=[b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.512028+0000", 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.354081976, 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065887159999999997, 2024-09-16T03:40:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 421 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512028+0000", 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512028+0000", 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512030+0000", 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1349999999999999e-06 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512036+0000", 2024-09-16T03:40:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8839999999999999e-06 2024-09-16T03:40:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512051+0000", 2024-09-16T03:40:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4698999999999999e-05 2024-09-16T03:40:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512058+0000", 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2540000000000003e-06 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512113+0000", 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5098e-05 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.512118+0000", 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.305e-06 2024-09-16T03:40:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.513450+0000", 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001331558 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.513661+0000", 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000210869 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514391+0000", 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073004199999999997 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518031+0000", 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003639626 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518526+0000", 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00049509499999999998 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518617+0000", 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1150999999999995e-05 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.bs4 54/31 ECSubWrite(tid=66, reqid=osd.10.0:66, at_version=54'30, trim_to=0'0, roll_forward_to=53'28))", 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.514028+0000", 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.352081733, 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00691765, 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514028+0000", 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514028+0000", 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514029+0000", 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8100000000000002e-07 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514047+0000", 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8420999999999999e-05 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514056+0000", 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7660000000000004e-06 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514063+0000", 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3829999999999997e-06 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514106+0000", 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3544999999999997e-05 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514113+0000", 2024-09-16T03:40:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5050000000000004e-06 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520899+0000", 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067864600000000002 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520946+0000", 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6888999999999998e-05 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 54/31 ECSubWrite(tid=62, reqid=osd.14.0:62, at_version=54'52, trim_to=0'0, roll_forward_to=53'48))", 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.514225+0000", 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.351884933999999, 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067025690000000002, 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514225+0000", 2024-09-16T03:40:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514225+0000", 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514226+0000", 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.82e-07 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514242+0000", 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6246e-05 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514250+0000", 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9659999999999996e-06 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514256+0000", 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9569999999999999e-06 2024-09-16T03:40:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514309+0000", 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3075999999999998e-05 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.514321+0000", 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2337999999999999e-05 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520884+0000", 2024-09-16T03:40:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065623139999999996 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520928+0000", 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.409e-05 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.3s4 54/31 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=54'16, trim_to=0'0, roll_forward_to=53'14))", 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.515287+0000", 2024-09-16T03:40:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.350822548, 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056953539999999997, 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515287+0000", 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515287+0000", 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515288+0000", 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6199999999999999e-07 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515311+0000", 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3591000000000002e-05 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515324+0000", 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2583e-05 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515335+0000", 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0883999999999999e-05 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515402+0000", 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7241000000000002e-05 2024-09-16T03:40:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.515412+0000", 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0482999999999999e-05 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520951+0000", 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055382469999999996 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.520983+0000", 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2162999999999997e-05 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s5 54/31 ECSubWrite(tid=57, reqid=osd.6.0:56, at_version=0'0, trim_to=0'0, roll_forward_to=54'45))", 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:33.518434+0000", 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.34767583, 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055100909999999999, 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518434+0000", 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518434+0000", 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518435+0000", 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3099999999999998e-07 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518445+0000", 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.043e-05 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518460+0000", 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5260999999999998e-05 2024-09-16T03:40:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518468+0000", 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2309999999999999e-06 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518527+0000", 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9704999999999998e-05 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.518536+0000", 2024-09-16T03:40:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3189999999999995e-06 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.523821+0000", 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052849619999999998 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:33.523944+0000", 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123752 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:460 3.0s0 3:08509fe0:::smithi12138290-16:head [append 0~458752] snapc 10=[10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:37.475645+0000", 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.390464535, 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059849059999999999, 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 460 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.475645+0000", 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.475645+0000", 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.475647+0000", 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.53e-06 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.476110+0000", 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00046279300000000003 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.476126+0000", 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6399999999999999e-05 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.476134+0000", 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5809999999999999e-06 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.476187+0000", 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3325999999999997e-05 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.476193+0000", 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3570000000000003e-06 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.477725+0000", 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001532013 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.477793+0000", 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7631000000000004e-05 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.478606+0000", 2024-09-16T03:40:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081251800000000003 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479831+0000", 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012256700000000001 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.481537+0000", 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001706068 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.481630+0000", 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3018999999999998e-05 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:461 3.0s0 3:08509fe0:::smithi12138290-16:head [append 0~688128] snapc 10=[10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:37.479182+0000", 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.386928418, 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053366230000000004, 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 461 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479182+0000", 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479182+0000", 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479183+0000", 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2470000000000001e-06 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479896+0000", 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071288299999999998 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479908+0000", 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2478e-05 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479916+0000", 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9589999999999995e-06 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.479966+0000", 2024-09-16T03:40:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9975999999999997e-05 2024-09-16T03:40:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.480029+0000", 2024-09-16T03:40:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3003999999999995e-05 2024-09-16T03:40:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.480645+0000", 2024-09-16T03:40:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061600999999999995 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.481642+0000", 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00099715099999999994 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484466+0000", 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028235180000000001 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484518+0000", 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2397e-05 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:463 3.0s0 3:08509fe0:::smithi12138290-16:head [setxattr _header (57) in=7b] snapc 10=[10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:37.482677+0000", 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.383432543, 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00709101, 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 463 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482677+0000", 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482677+0000", 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482678+0000", 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4199999999999995e-07 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482681+0000", 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0290000000000001e-06 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482686+0000", 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.164e-06 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482698+0000", 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1635e-05 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.483801+0000", 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001103436 2024-09-16T03:40:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.483877+0000", 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5174999999999998e-05 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484163+0000", 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00028641499999999999 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.486805+0000", 2024-09-16T03:40:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026422709999999999 2024-09-16T03:40:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:40:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.489716+0000", 2024-09-16T03:40:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029110749999999999 2024-09-16T03:40:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.489768+0000", 2024-09-16T03:40:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2167999999999999e-05 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:464 3.0s0 3:08509fe0:::smithi12138290-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:37.482705+0000", 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.383404649999999, 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0081540769999999992, 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:40:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 464 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482705+0000", 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482705+0000", 2024-09-16T03:40:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482706+0000", 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2800000000000002e-07 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482713+0000", 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.759e-06 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482719+0000", 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.482722+0000", 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1219999999999999e-06 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484586+0000", 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001864386 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.484638+0000", 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1718000000000002e-05 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.490065+0000", 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054270029999999997 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.490116+0000", 2024-09-16T03:40:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0519999999999997e-05 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:37.490859+0000", 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00074374800000000002 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:40:48.978 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:48.979 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:48.980 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:48.978+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:48.980 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:48.981 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:48.981 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:48.981 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:40:48.982 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:48.982 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:48.999 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:48.999 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:49.015 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:49.015 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:49.032 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:49.032 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:49.049 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:49.049 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:49.065 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:49.065 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:49.080 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:49.077+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:49.081 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:49.081 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:49.097 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:49.097 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:49.114 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:49.115 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:49.125 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:49.125 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:49.131 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:49.131 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:49.133 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:40:49.147 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:49.147 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:49.148 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-09-16T03:40:49.164 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:49.164 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:49.162+0000 7fe411198640 0 -- 172.21.15.76:0/3869365785 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7fe40c21e900 msgr2=0x7fe40c21ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:49.181 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:49.179+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:49.244 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:40:49.282 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:49.281+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:49.382 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:49.382 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:49.383 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:49.380+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:40:49.385 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:40:49.387 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:49.388 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:49.397 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:40:49.483 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:49.481+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:49.510 INFO:tasks.rados.rados.0.smithi121.stdout:232: done (4 left) 2024-09-16T03:40:49.510 INFO:tasks.rados.rados.0.smithi121.stdout:233: setattr oid 18 current snap is 23 2024-09-16T03:40:49.510 INFO:tasks.rados.rados.0.smithi121.stdout:228: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:40:49.526 INFO:tasks.rados.rados.0.smithi121.stdout:229: finishing write tid 1 to smithi12138290-41 2024-09-16T03:40:49.526 INFO:tasks.rados.rados.0.smithi121.stdout:229: finishing write tid 2 to smithi12138290-41 2024-09-16T03:40:49.526 INFO:tasks.rados.rados.0.smithi121.stdout:229: finishing write tid 3 to smithi12138290-41 2024-09-16T03:40:49.526 INFO:tasks.rados.rados.0.smithi121.stdout:229: finishing write tid 4 to smithi12138290-41 2024-09-16T03:40:49.526 INFO:tasks.rados.rados.0.smithi121.stdout:229: finishing write tid 5 to smithi12138290-41 2024-09-16T03:40:49.526 INFO:tasks.rados.rados.0.smithi121.stdout:229: oid 41 updating version 0 to 35 2024-09-16T03:40:49.527 INFO:tasks.rados.rados.0.smithi121.stdout:229: oid 41 updating version 35 to 38 2024-09-16T03:40:49.527 INFO:tasks.rados.rados.0.smithi121.stdout:229: oid 41 version 38 is already newer than 37 2024-09-16T03:40:49.527 INFO:tasks.rados.rados.0.smithi121.stdout:229: oid 41 version 38 is already newer than 36 2024-09-16T03:40:49.527 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 41 v 38 (ObjNum 111 snap 23 seq_num 111) dirty exists 2024-09-16T03:40:49.527 INFO:tasks.rados.rados.0.smithi121.stdout:229: left oid 41 (ObjNum 111 snap 23 seq_num 111) 2024-09-16T03:40:49.527 INFO:tasks.rados.rados.0.smithi121.stdout:230: finishing write tid 1 to smithi12138290-36 2024-09-16T03:40:49.527 INFO:tasks.rados.rados.0.smithi121.stdout:230: finishing write tid 2 to smithi12138290-36 2024-09-16T03:40:49.527 INFO:tasks.rados.rados.0.smithi121.stdout:230: finishing write tid 3 to smithi12138290-36 2024-09-16T03:40:49.527 INFO:tasks.rados.rados.0.smithi121.stdout:230: finishing write tid 4 to smithi12138290-36 2024-09-16T03:40:49.527 INFO:tasks.rados.rados.0.smithi121.stdout:230: finishing write tid 5 to smithi12138290-36 2024-09-16T03:40:49.528 INFO:tasks.rados.rados.0.smithi121.stdout:230: oid 36 updating version 0 to 13 2024-09-16T03:40:49.528 INFO:tasks.rados.rados.0.smithi121.stdout:230: oid 36 version 13 is already newer than 12 2024-09-16T03:40:49.528 INFO:tasks.rados.rados.0.smithi121.stdout:230: oid 36 version 13 is already newer than 11 2024-09-16T03:40:49.528 INFO:tasks.rados.rados.0.smithi121.stdout:230: oid 36 version 13 is already newer than 10 2024-09-16T03:40:49.528 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 36 v 13 (ObjNum 112 snap 23 seq_num 112) dirty exists 2024-09-16T03:40:49.528 INFO:tasks.rados.rados.0.smithi121.stdout:230: left oid 36 (ObjNum 112 snap 23 seq_num 112) 2024-09-16T03:40:49.528 INFO:tasks.rados.rados.0.smithi121.stdout:231: finishing write tid 1 to smithi12138290-13 2024-09-16T03:40:49.528 INFO:tasks.rados.rados.0.smithi121.stdout:228: done (4 left) 2024-09-16T03:40:49.528 INFO:tasks.rados.rados.0.smithi121.stdout:229: done (3 left) 2024-09-16T03:40:49.528 INFO:tasks.rados.rados.0.smithi121.stdout:230: done (2 left) 2024-09-16T03:40:49.529 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 32 current snap is 23 2024-09-16T03:40:49.529 INFO:tasks.rados.rados.0.smithi121.stdout:234: seq_num 114 ranges {0=753664,753664=688128,1441792=475136} 2024-09-16T03:40:49.529 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:40:49.530 INFO:tasks.rados.rados.0.smithi121.stdout:234: writing smithi12138290-32 from 0 to 753664 tid 1 2024-09-16T03:40:49.534 INFO:tasks.rados.rados.0.smithi121.stdout:234: writing smithi12138290-32 from 753664 to 1441792 tid 2 2024-09-16T03:40:49.539 INFO:tasks.rados.rados.0.smithi121.stdout:234: writing smithi12138290-32 from 1441792 to 1916928 tid 3 2024-09-16T03:40:49.539 INFO:tasks.rados.rados.0.smithi121.stdout:235: copy_from oid 5 from oid 38 current snap is 23 2024-09-16T03:40:49.539 INFO:tasks.rados.rados.0.smithi121.stdout:236: read oid 44 snap -1 2024-09-16T03:40:49.539 INFO:tasks.rados.rados.0.smithi121.stdout:236: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-16T03:40:49.539 INFO:tasks.rados.rados.0.smithi121.stdout:237: copy_from oid 10 from oid 23 current snap is 23 2024-09-16T03:40:49.540 INFO:tasks.rados.rados.0.smithi121.stdout:238: snap_create 2024-09-16T03:40:49.540 INFO:tasks.rados.rados.0.smithi121.stdout:231: finishing write tid 2 to smithi12138290-13 2024-09-16T03:40:49.540 INFO:tasks.rados.rados.0.smithi121.stdout:231: finishing write tid 3 to smithi12138290-13 2024-09-16T03:40:49.540 INFO:tasks.rados.rados.0.smithi121.stdout:231: finishing write tid 4 to smithi12138290-13 2024-09-16T03:40:49.540 INFO:tasks.rados.rados.0.smithi121.stdout:231: finishing write tid 5 to smithi12138290-13 2024-09-16T03:40:49.541 INFO:tasks.rados.rados.0.smithi121.stdout:231: oid 13 updating version 0 to 42 2024-09-16T03:40:49.541 INFO:tasks.rados.rados.0.smithi121.stdout:231: oid 13 version 42 is already newer than 39 2024-09-16T03:40:49.541 INFO:tasks.rados.rados.0.smithi121.stdout:231: oid 13 version 42 is already newer than 40 2024-09-16T03:40:49.541 INFO:tasks.rados.rados.0.smithi121.stdout:231: oid 13 version 42 is already newer than 41 2024-09-16T03:40:49.541 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 13 v 42 (ObjNum 113 snap 23 seq_num 113) dirty exists 2024-09-16T03:40:49.541 INFO:tasks.rados.rados.0.smithi121.stdout:231: left oid 13 (ObjNum 113 snap 23 seq_num 113) 2024-09-16T03:40:49.542 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 18 v 69 (ObjNum 89 snap 12 seq_num 89) dirty exists 2024-09-16T03:40:49.543 INFO:tasks.rados.rados.0.smithi121.stdout:235: finishing copy_from to smithi12138290-5 2024-09-16T03:40:49.543 INFO:tasks.rados.rados.0.smithi121.stdout:235: got expected ENOENT (src dne) 2024-09-16T03:40:49.543 INFO:tasks.rados.rados.0.smithi121.stdout:235: finishing copy_from racing read to smithi12138290-5 2024-09-16T03:40:49.545 INFO:tasks.rados.rados.0.smithi121.stdout:234: finishing write tid 1 to smithi12138290-32 2024-09-16T03:40:49.546 INFO:tasks.rados.rados.0.smithi121.stdout:234: finishing write tid 2 to smithi12138290-32 2024-09-16T03:40:49.546 INFO:tasks.rados.rados.0.smithi121.stdout:236: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-16T03:40:49.547 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:49.547 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:49.558 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:40:49.566 INFO:tasks.rados.rados.0.smithi121.stdout:234: finishing write tid 3 to smithi12138290-32 2024-09-16T03:40:49.566 INFO:tasks.rados.rados.0.smithi121.stdout:234: finishing write tid 4 to smithi12138290-32 2024-09-16T03:40:49.566 INFO:tasks.rados.rados.0.smithi121.stdout:234: finishing write tid 5 to smithi12138290-32 2024-09-16T03:40:49.566 INFO:tasks.rados.rados.0.smithi121.stdout:234: oid 32 updating version 0 to 76 2024-09-16T03:40:49.566 INFO:tasks.rados.rados.0.smithi121.stdout:234: oid 32 version 76 is already newer than 74 2024-09-16T03:40:49.566 INFO:tasks.rados.rados.0.smithi121.stdout:234: oid 32 version 76 is already newer than 69 2024-09-16T03:40:49.566 INFO:tasks.rados.rados.0.smithi121.stdout:234: oid 32 version 76 is already newer than 75 2024-09-16T03:40:49.566 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 32 v 76 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-09-16T03:40:49.567 INFO:tasks.rados.rados.0.smithi121.stdout:234: left oid 32 (ObjNum 114 snap 23 seq_num 114) 2024-09-16T03:40:49.567 INFO:tasks.rados.rados.0.smithi121.stdout:237: finishing copy_from racing read to smithi12138290-10 2024-09-16T03:40:49.567 INFO:tasks.rados.rados.0.smithi121.stdout:237: finishing copy_from to smithi12138290-10 2024-09-16T03:40:49.567 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 10 v 16 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-09-16T03:40:49.584 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:49.582+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:49.662 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:49.662 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:49.671 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:40:49.714 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:49.712+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:49.768 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:49.766+0000 7f15a020c640 -1 Falling back to public interface 2024-09-16T03:40:49.779 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:49.779 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:49.785 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:49.783+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:49.788 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:40:49.887 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:49.885+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:49.904 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:49.904 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:49.913 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:40:49.987 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:49.986+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:50.059 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:50.059 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:50.068 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:40:50.088 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:50.086+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:50.172 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:50.172 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:50.181 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:40:50.189 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:50.186+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:50.285 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:50.285 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:50.290 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:50.288+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:50.293 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:40:50.391 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:50.389+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:50.395 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:50.395 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:50.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:40:50.492 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:50.490+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:50.508 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:50.508 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:50.511 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T03:40:50.514 INFO:tasks.rados.rados.0.smithi121.stdout:231: done (6 left) 2024-09-16T03:40:50.514 INFO:tasks.rados.rados.0.smithi121.stdout:233: done (5 left) 2024-09-16T03:40:50.514 INFO:tasks.rados.rados.0.smithi121.stdout:234: done (4 left) 2024-09-16T03:40:50.514 INFO:tasks.rados.rados.0.smithi121.stdout:235: done (3 left) 2024-09-16T03:40:50.514 INFO:tasks.rados.rados.0.smithi121.stdout:236: done (2 left) 2024-09-16T03:40:50.514 INFO:tasks.rados.rados.0.smithi121.stdout:237: done (1 left) 2024-09-16T03:40:50.514 INFO:tasks.rados.rados.0.smithi121.stdout:238: done (0 left) 2024-09-16T03:40:50.514 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 21 current snap is 24 2024-09-16T03:40:50.514 INFO:tasks.rados.rados.0.smithi121.stdout:239: seq_num 115 ranges {1785856=770048,2555904=704512,3260416=458752} 2024-09-16T03:40:50.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:40:50.527 INFO:tasks.rados.rados.0.smithi121.stdout:239: writing smithi12138290-21 from 1785856 to 2555904 tid 1 2024-09-16T03:40:50.531 INFO:tasks.rados.rados.0.smithi121.stdout:239: writing smithi12138290-21 from 2555904 to 3260416 tid 2 2024-09-16T03:40:50.534 INFO:tasks.rados.rados.0.smithi121.stdout:239: writing smithi12138290-21 from 3260416 to 3719168 tid 3 2024-09-16T03:40:50.534 INFO:tasks.rados.rados.0.smithi121.stdout:240: snap_remove snap 19 2024-09-16T03:40:50.593 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:50.591+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:50.660 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:50.660 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:50.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:40:50.713 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:50.711+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:50.772 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:50.773 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:50.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:40:50.823 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:50.821+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:50.896 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:50.894+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:50.925 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:50.926 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:50.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:40:50.997 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:50.995+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:51.078 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:51.078 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:51.088 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:40:51.098 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:51.096+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:51.192 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:51.192 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:51.198 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:51.197+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:51.200 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:40:51.300 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:51.297+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:51.343 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:51.343 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:51.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:40:51.387 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:51.431 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:51.429+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:51.467 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:51.467 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:51.475 INFO:tasks.rados.rados.0.smithi121.stdout:240: done (1 left) 2024-09-16T03:40:51.475 INFO:tasks.rados.rados.0.smithi121.stdout:241: read oid 44 snap -1 2024-09-16T03:40:51.475 INFO:tasks.rados.rados.0.smithi121.stdout:241: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-16T03:40:51.475 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 1 current snap is 24 2024-09-16T03:40:51.475 INFO:tasks.rados.rados.0.smithi121.stdout:239: finishing write tid 1 to smithi12138290-21 2024-09-16T03:40:51.475 INFO:tasks.rados.rados.0.smithi121.stdout:242: seq_num 116 ranges {0=786432,786432=458752,1245184=442368,1687552=262144} 2024-09-16T03:40:51.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:40:51.481 INFO:tasks.rados.rados.0.smithi121.stdout:242: writing smithi12138290-1 from 0 to 786432 tid 1 2024-09-16T03:40:51.483 INFO:tasks.rados.rados.0.smithi121.stdout:242: writing smithi12138290-1 from 786432 to 1245184 tid 2 2024-09-16T03:40:51.486 INFO:tasks.rados.rados.0.smithi121.stdout:242: writing smithi12138290-1 from 1245184 to 1687552 tid 3 2024-09-16T03:40:51.488 INFO:tasks.rados.rados.0.smithi121.stdout:242: writing smithi12138290-1 from 1687552 to 1949696 tid 4 2024-09-16T03:40:51.489 INFO:tasks.rados.rados.0.smithi121.stdout:243: rollback oid 34 current snap is 24 2024-09-16T03:40:51.489 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 34 to 22 2024-09-16T03:40:51.489 INFO:tasks.rados.rados.0.smithi121.stdout:239: finishing write tid 2 to smithi12138290-21 2024-09-16T03:40:51.489 INFO:tasks.rados.rados.0.smithi121.stdout:239: finishing write tid 3 to smithi12138290-21 2024-09-16T03:40:51.489 INFO:tasks.rados.rados.0.smithi121.stdout:239: finishing write tid 4 to smithi12138290-21 2024-09-16T03:40:51.489 INFO:tasks.rados.rados.0.smithi121.stdout:239: finishing write tid 5 to smithi12138290-21 2024-09-16T03:40:51.489 INFO:tasks.rados.rados.0.smithi121.stdout:239: oid 21 updating version 0 to 24 2024-09-16T03:40:51.490 INFO:tasks.rados.rados.0.smithi121.stdout:239: oid 21 version 24 is already newer than 23 2024-09-16T03:40:51.490 INFO:tasks.rados.rados.0.smithi121.stdout:239: oid 21 version 24 is already newer than 22 2024-09-16T03:40:51.490 INFO:tasks.rados.rados.0.smithi121.stdout:239: oid 21 version 24 is already newer than 21 2024-09-16T03:40:51.490 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 21 v 24 (ObjNum 115 snap 24 seq_num 115) dirty exists 2024-09-16T03:40:51.490 INFO:tasks.rados.rados.0.smithi121.stdout:239: left oid 21 (ObjNum 115 snap 24 seq_num 115) 2024-09-16T03:40:51.490 INFO:tasks.rados.rados.0.smithi121.stdout:239: done (3 left) 2024-09-16T03:40:51.490 INFO:tasks.rados.rados.0.smithi121.stdout:244: rollback oid 26 current snap is 24 2024-09-16T03:40:51.490 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 26 to 20 2024-09-16T03:40:51.490 INFO:tasks.rados.rados.0.smithi121.stdout:245: snap_remove snap 3 2024-09-16T03:40:51.501 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:51.498+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:51.533 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:51.533 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:51.551 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:51.551 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:51.567 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:51.567 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:51.584 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:51.584 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:51.588 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:51.589 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:51.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:40:51.599 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:51.601 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:51.599+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:51.616 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:51.616 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:51.633 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:51.633 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:51.649 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:51.665 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:51.665 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:51.682 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:51.682 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:51.698 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:51.699 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:51.703 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:51.701+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:51.715 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:51.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:51.731 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:51.731 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-09-16T03:40:51.742 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:51.742 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:51.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 42/31 ECSubWrite(tid=20, reqid=client.4615.0:251, at_version=42'11, trim_to=0'0, roll_forward_to=34'9))", 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.564293+0000", 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.175654770000001, 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044757829999999997, 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564293+0000", 2024-09-16T03:40:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564293+0000", 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564293+0000", 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.53e-07 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564405+0000", 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111525 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564417+0000", 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2046e-05 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564424+0000", 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2760000000000003e-06 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564521+0000", 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7120999999999994e-05 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.564532+0000", 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0419e-05 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.568719+0000", 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041873969999999998 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.568769+0000", 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9746000000000001e-05 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 42/31 ECSubWrite(tid=24, reqid=client.4615.0:268, at_version=42'15, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.629096+0000", 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.110851746000002, 2024-09-16T03:40:51.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027011040000000002, 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629096+0000", 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629096+0000", 2024-09-16T03:40:51.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629097+0000", 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5700000000000002e-07 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629153+0000", 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5983000000000003e-05 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629163+0000", 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0135e-05 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629171+0000", 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1210000000000007e-06 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629247+0000", 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6074000000000004e-05 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629257+0000", 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0098000000000001e-05 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631773+0000", 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025155759999999998 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631797+0000", 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.446e-05 2024-09-16T03:40:51.751 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 42/31 ECSubWrite(tid=25, reqid=client.4615.0:269, at_version=42'16, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.629979+0000", 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.109968310999999, 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045141219999999998, 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629979+0000", 2024-09-16T03:40:51.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629979+0000", 2024-09-16T03:40:51.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629980+0000", 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.95e-07 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629998+0000", 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7475000000000001e-05 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630012+0000", 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3737000000000001e-05 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630020+0000", 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4239999999999993e-06 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630068+0000", 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7729000000000003e-05 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630073+0000", 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5090000000000001e-06 2024-09-16T03:40:51.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634467+0000", 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043935040000000003 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634493+0000", 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6749000000000001e-05 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 43/31 ECSubWrite(tid=34, reqid=client.4615.0:282, at_version=43'6, trim_to=0'0, roll_forward_to=34'4))", 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:22.579052+0000", 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.160895467, 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002945193, 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579052+0000", 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579052+0000", 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579052+0000", 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7499999999999999e-07 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579466+0000", 2024-09-16T03:40:51.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000413664 2024-09-16T03:40:51.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579484+0000", 2024-09-16T03:40:51.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8238999999999999e-05 2024-09-16T03:40:51.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579493+0000", 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7600000000000008e-06 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579570+0000", 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7218999999999994e-05 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.579581+0000", 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0599e-05 2024-09-16T03:40:51.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581952+0000", 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002371536 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:22.581997+0000", 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5000999999999998e-05 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/31 ECSubWrite(tid=30, reqid=client.4615.0:301, at_version=49'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.487167+0000", 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.252781024000001, 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005489954, 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487167+0000", 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487167+0000", 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487167+0000", 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4900000000000001e-07 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487497+0000", 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032991100000000001 2024-09-16T03:40:51.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487511+0000", 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3652000000000001e-05 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487522+0000", 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1224e-05 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487609+0000", 2024-09-16T03:40:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6839e-05 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487627+0000", 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8173e-05 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492620+0000", 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049931430000000002 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492657+0000", 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6563000000000002e-05 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/31 ECSubWrite(tid=31, reqid=osd.6.0:31, at_version=49'22, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.489137+0000", 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.250810513000001, 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052790479999999997, 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489137+0000", 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489137+0000", 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489138+0000", 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1090000000000001e-06 2024-09-16T03:40:51.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489291+0000", 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015225699999999999 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489309+0000", 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8943000000000001e-05 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489320+0000", 2024-09-16T03:40:51.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0458e-05 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489403+0000", 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2935999999999993e-05 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489414+0000", 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1548e-05 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494378+0000", 2024-09-16T03:40:51.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049636970000000004 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494416+0000", 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8099999999999998e-05 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/31 ECSubWrite(tid=32, reqid=client.4615.0:302, at_version=49'23, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.490769+0000", 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.249178397000001, 2024-09-16T03:40:51.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047418249999999999, 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490769+0000", 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490769+0000", 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490770+0000", 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.083e-06 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491234+0000", 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046313399999999997 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491247+0000", 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3773e-05 2024-09-16T03:40:51.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491257+0000", 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4439999999999997e-06 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491324+0000", 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7779000000000003e-05 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491334+0000", 2024-09-16T03:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9990000000000003e-06 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495481+0000", 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041461240000000002 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495511+0000", 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0488999999999999e-05 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/31 ECSubWrite(tid=33, reqid=client.4615.0:303, at_version=49'24, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.492676+0000", 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.247271865999998, 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028790970000000002, 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492676+0000", 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492676+0000", 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492677+0000", 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9999999999999997e-07 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492696+0000", 2024-09-16T03:40:51.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9471999999999999e-05 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492709+0000", 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3193999999999999e-05 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492718+0000", 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3190000000000001e-06 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492794+0000", 2024-09-16T03:40:51.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5196000000000003e-05 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492803+0000", 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9840000000000007e-06 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495527+0000", 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027244470000000001 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495555+0000", 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7784999999999999e-05 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/31 ECSubWrite(tid=34, reqid=client.4615.0:303, at_version=0'0, trim_to=0'0, roll_forward_to=49'24))", 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.502221+0000", 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.237726290000001, 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047398429999999997, 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.779 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502221+0000", 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502221+0000", 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502222+0000", 2024-09-16T03:40:51.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502232+0000", 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0001e-05 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502245+0000", 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2889e-05 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502253+0000", 2024-09-16T03:40:51.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0979999999999994e-06 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502341+0000", 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8129999999999998e-05 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502353+0000", 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.203e-05 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.506922+0000", 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045686820000000001 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.506961+0000", 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9332999999999998e-05 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 49/31 ECSubWrite(tid=44, reqid=osd.10.0:44, at_version=49'11, trim_to=0'0, roll_forward_to=43'9))", 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.502722+0000", 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.237226054000001, 2024-09-16T03:40:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042510680000000002, 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502722+0000", 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502722+0000", 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502722+0000", 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5300000000000003e-07 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502747+0000", 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4915000000000001e-05 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502771+0000", 2024-09-16T03:40:51.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3830000000000001e-05 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502781+0000", 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7010000000000008e-06 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502843+0000", 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2323000000000006e-05 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502853+0000", 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.893e-06 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.506929+0000", 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040762910000000001 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.506973+0000", 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3262e-05 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/31 ECSubWrite(tid=35, reqid=client.4615.0:309, at_version=49'26, trim_to=0'0, roll_forward_to=49'24))", 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.503982+0000", 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.235965504999999, 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054062880000000004, 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.503982+0000", 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.503982+0000", 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.503983+0000", 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2200000000000003e-07 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504266+0000", 2024-09-16T03:40:51.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000282949 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504276+0000", 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9569999999999997e-06 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504284+0000", 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2009999999999999e-06 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504344+0000", 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9670000000000003e-05 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504353+0000", 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2900000000000008e-06 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.509351+0000", 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049981820000000003 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.509388+0000", 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7217e-05 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 52/31 ECSubWrite(tid=23, reqid=client.4615.0:332, at_version=52'12, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:31.478445+0000", 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.261502918000001, 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028330030000000002, 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.792 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.478445+0000", 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.478445+0000", 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.478445+0000", 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3300000000000003e-07 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.478999+0000", 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055398199999999996 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.479016+0000", 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6458999999999999e-05 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.479030+0000", 2024-09-16T03:40:51.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4616e-05 2024-09-16T03:40:51.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.479114+0000", 2024-09-16T03:40:51.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3469999999999999e-05 2024-09-16T03:40:51.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.479123+0000", 2024-09-16T03:40:51.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6339999999999996e-06 2024-09-16T03:40:51.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.481233+0000", 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021093359999999998 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.481278+0000", 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5073e-05 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.797 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 52/31 ECSubWrite(tid=24, reqid=client.4615.0:333, at_version=52'13, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:31.479858+0000", 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.260089826000002, 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003878116, 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.479858+0000", 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.479858+0000", 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.479859+0000", 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099e-06 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.480178+0000", 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000318745 2024-09-16T03:40:51.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.480189+0000", 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1416e-05 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.480200+0000", 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0597999999999999e-05 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.480312+0000", 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112495 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.480323+0000", 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0645999999999999e-05 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.483684+0000", 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033609529999999999 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.483736+0000", 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2163999999999997e-05 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 52/31 ECSubWrite(tid=25, reqid=client.4615.0:334, at_version=52'14, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:31.481284+0000", 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.258663643999999, 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029360279999999998, 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.802 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.481284+0000", 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.481284+0000", 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.481285+0000", 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.481357+0000", 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2434000000000005e-05 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.481369+0000", 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1841999999999999e-05 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.481380+0000", 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1420000000000001e-05 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.481451+0000", 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0393000000000002e-05 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.481460+0000", 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4550000000000001e-06 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.484186+0000", 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027256279999999999 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.484220+0000", 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4227000000000003e-05 2024-09-16T03:40:51.806 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 52/31 ECSubWrite(tid=45, reqid=client.4615.0:338, at_version=52'37, trim_to=0'0, roll_forward_to=51'34))", 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:31.498676+0000", 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.241271432000001, 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002907129, 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.498676+0000", 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.498676+0000", 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.498677+0000", 2024-09-16T03:40:51.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1119999999999999e-06 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499293+0000", 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061556100000000004 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499312+0000", 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8612999999999999e-05 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499322+0000", 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0278999999999999e-05 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499393+0000", 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.093e-05 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499403+0000", 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0604999999999999e-05 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.501543+0000", 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002139195 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.501583+0000", 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0834000000000002e-05 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.811 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 52/31 ECSubWrite(tid=46, reqid=client.4615.0:339, at_version=52'38, trim_to=0'0, roll_forward_to=51'34))", 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:31.499639+0000", 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.240308711000001, 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045842979999999997, 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499639+0000", 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499639+0000", 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499640+0000", 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4099999999999998e-07 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499710+0000", 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9930000000000003e-05 2024-09-16T03:40:51.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499722+0000", 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2728000000000001e-05 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499732+0000", 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3510000000000008e-06 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499901+0000", 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016893999999999999 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.499911+0000", 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.504180+0000", 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042690109999999996 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.504223+0000", 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3084000000000003e-05 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 52/31 ECSubWrite(tid=47, reqid=client.4615.0:340, at_version=52'39, trim_to=0'0, roll_forward_to=51'34))", 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:31.500450+0000", 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.239498001000001, 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003811512, 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.816 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.500450+0000", 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.500450+0000", 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.500450+0000", 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7000000000000004e-07 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.500466+0000", 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5159e-05 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.500477+0000", 2024-09-16T03:40:51.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1447000000000001e-05 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.500485+0000", 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8639999999999996e-06 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.500540+0000", 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4982000000000003e-05 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.500546+0000", 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2589999999999997e-06 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.504229+0000", 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036825540000000002 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.504261+0000", 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2477000000000001e-05 2024-09-16T03:40:51.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 58/31 ECSubWrite(tid=51, reqid=client.4615.0:460, at_version=58'20, trim_to=0'0, roll_forward_to=55'18))", 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:37.478499+0000", 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.261448929, 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026724280000000001, 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.478499+0000", 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.478499+0000", 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.478499+0000", 2024-09-16T03:40:51.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4299999999999999e-07 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.478719+0000", 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021992600000000001 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.478735+0000", 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6266999999999999e-05 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.478745+0000", 2024-09-16T03:40:51.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2920000000000001e-06 2024-09-16T03:40:51.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.478929+0000", 2024-09-16T03:40:51.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000184296 2024-09-16T03:40:51.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.478943+0000", 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4361e-05 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.481130+0000", 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002186805 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.481171+0000", 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1137999999999998e-05 2024-09-16T03:40:51.825 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 58/31 ECSubWrite(tid=52, reqid=client.4615.0:461, at_version=58'21, trim_to=0'0, roll_forward_to=55'18))", 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:37.480643+0000", 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.259304672000001, 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026947070000000002, 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.480643+0000", 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.480643+0000", 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.480644+0000", 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7900000000000007e-07 2024-09-16T03:40:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.481184+0000", 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054000599999999999 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.481198+0000", 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4259e-05 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.481208+0000", 2024-09-16T03:40:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0094e-05 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.481321+0000", 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011300500000000001 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.481328+0000", 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5289999999999997e-06 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.483302+0000", 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001973762 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:37.483338+0000", 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6072999999999998e-05 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 68/31 ECSubWrite(tid=83, reqid=client.4615.0:584, at_version=68'64, trim_to=0'0, roll_forward_to=66'62))", 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:47.490050+0000", 2024-09-16T03:40:51.830 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.2498976949999996, 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002854746, 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:47.490050+0000", 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:47.490050+0000", 2024-09-16T03:40:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:47.490050+0000", 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8799999999999999e-07 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:47.490071+0000", 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0684999999999999e-05 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:51.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:47.490082+0000", 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1388e-05 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:47.490091+0000", 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4819999999999994e-06 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:47.490165+0000", 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4388000000000007e-05 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:51.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:47.490180+0000", 2024-09-16T03:40:51.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.504e-05 2024-09-16T03:40:51.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:51.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:47.492869+0000", 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026891649999999999 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:47.492905+0000", 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5410000000000001e-05 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:51.836 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:51.838 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:51.838 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:51.836+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:51.839 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:51.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:40:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:40:51.840 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:51.841 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:40:51.938 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:51.937+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:51.982 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:51.982 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:51.982 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:51.982 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:51.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:40:52.040 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:52.039+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:52.093 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:52.093 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:52.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:40:52.141 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:52.139+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:52.205 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 37/31 ECSubWrite(tid=23, reqid=client.4615.0:205, at_version=37'14, trim_to=0'0, roll_forward_to=34'12))", 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 35.678061866, 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050337250000000002, 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.525220+0000", 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9999999999999999e-07 2024-09-16T03:40:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526147+0000", 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00092697700000000005 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526170+0000", 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3377e-05 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526182+0000", 2024-09-16T03:40:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1331999999999999e-05 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526338+0000", 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000156009 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.526347+0000", 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7610000000000002e-06 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530193+0000", 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003845209 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:16.530254+0000", 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0859999999999997e-05 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 42/31 ECSubWrite(tid=30, reqid=osd.14.0:30, at_version=42'20, trim_to=0'0, roll_forward_to=41'18))", 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.557656+0000", 2024-09-16T03:40:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.645625766999999, 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047645880000000002, 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557656+0000", 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557656+0000", 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557657+0000", 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.68e-07 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557674+0000", 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7887e-05 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557696+0000", 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2035000000000002e-05 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557705+0000", 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5750000000000001e-06 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557774+0000", 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9015000000000002e-05 2024-09-16T03:40:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.557785+0000", 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1289e-05 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562388+0000", 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004602783 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562420+0000", 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2335999999999997e-05 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 42/31 ECSubWrite(tid=74, reqid=osd.0.0:73, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.562839+0000", 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.640442177000001, 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047921420000000001, 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.215 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562839+0000", 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562839+0000", 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562840+0000", 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9899999999999999e-07 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562852+0000", 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1975e-05 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562863+0000", 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1100999999999999e-05 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562869+0000", 2024-09-16T03:40:52.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9139999999999996e-06 2024-09-16T03:40:52.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562921+0000", 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1564999999999998e-05 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.562927+0000", 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.455e-06 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.567594+0000", 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046666099999999999 2024-09-16T03:40:52.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.567632+0000", 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7722999999999998e-05 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 42/31 ECSubWrite(tid=32, reqid=osd.14.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.639556330000001, 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075866950000000001, 2024-09-16T03:40:52.220 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563726+0000", 2024-09-16T03:40:52.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.09e-07 2024-09-16T03:40:52.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563741+0000", 2024-09-16T03:40:52.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5123000000000001e-05 2024-09-16T03:40:52.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563761+0000", 2024-09-16T03:40:52.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9335999999999999e-05 2024-09-16T03:40:52.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563770+0000", 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3770000000000002e-06 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563840+0000", 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9725000000000001e-05 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563851+0000", 2024-09-16T03:40:52.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1107000000000001e-05 2024-09-16T03:40:52.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571277+0000", 2024-09-16T03:40:52.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074264830000000002 2024-09-16T03:40:52.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571312+0000", 2024-09-16T03:40:52.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4635000000000003e-05 2024-09-16T03:40:52.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.226 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/31 ECSubWrite(tid=24, reqid=client.4615.0:268, at_version=42'15, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.629148+0000", 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.574133336999999, 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053360220000000002, 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629148+0000", 2024-09-16T03:40:52.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629148+0000", 2024-09-16T03:40:52.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629149+0000", 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8400000000000003e-07 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629190+0000", 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0806999999999997e-05 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629227+0000", 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6551000000000003e-05 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629236+0000", 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7769999999999997e-06 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629292+0000", 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5980999999999998e-05 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629300+0000", 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4909999999999999e-06 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634449+0000", 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051495170000000002 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634484+0000", 2024-09-16T03:40:52.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5014000000000001e-05 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/31 ECSubWrite(tid=25, reqid=client.4615.0:269, at_version=42'16, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.573284947000001, 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004694803, 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.232 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.629997+0000", 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4900000000000005e-07 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630013+0000", 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5722e-05 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630025+0000", 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2234e-05 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.630033+0000", 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3560000000000004e-06 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631638+0000", 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00160494 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.631647+0000", 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4960000000000002e-06 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634669+0000", 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003022247 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.634692+0000", 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2059e-05 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.236 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=30, reqid=client.4615.0:301, at_version=49'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.716167260999999, 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083935009999999994, 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487115+0000", 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.36e-07 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487566+0000", 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045117600000000002 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487581+0000", 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5092000000000001e-05 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487591+0000", 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.058e-05 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487677+0000", 2024-09-16T03:40:52.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5635000000000003e-05 2024-09-16T03:40:52.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487685+0000", 2024-09-16T03:40:52.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2830000000000001e-06 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495470+0000", 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077842739999999999 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495508+0000", 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8225000000000001e-05 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.242 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 49/31 ECSubWrite(tid=34, reqid=client.4615.0:305, at_version=49'22, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:52.242 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.489031+0000", 2024-09-16T03:40:52.242 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.714250304, 2024-09-16T03:40:52.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064300850000000003, 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489031+0000", 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489031+0000", 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489032+0000", 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4700000000000002e-07 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489060+0000", 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000002e-05 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489076+0000", 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6013000000000001e-05 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489086+0000", 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9499999999999996e-06 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489306+0000", 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022061699999999999 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489317+0000", 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0895e-05 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495421+0000", 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061040540000000003 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495461+0000", 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0139000000000003e-05 2024-09-16T03:40:52.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=31, reqid=osd.6.0:31, at_version=49'22, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.489235+0000", 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.714046198999998, 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0063054690000000002, 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489235+0000", 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489235+0000", 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489236+0000", 2024-09-16T03:40:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5499999999999996e-07 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489261+0000", 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5045000000000002e-05 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489275+0000", 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3743e-05 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489283+0000", 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2109999999999998e-06 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490215+0000", 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093184699999999997 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490227+0000", 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1737e-05 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495519+0000", 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052920049999999998 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495541+0000", 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1926e-05 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.252 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 49/31 ECSubWrite(tid=35, reqid=client.4615.0:306, at_version=49'23, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.712964873000001, 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065860449999999996, 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490318+0000", 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5600000000000004e-07 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490342+0000", 2024-09-16T03:40:52.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4006e-05 2024-09-16T03:40:52.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490356+0000", 2024-09-16T03:40:52.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4449e-05 2024-09-16T03:40:52.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490366+0000", 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8689999999999999e-06 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494506+0000", 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041400029999999997 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494516+0000", 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0229e-05 2024-09-16T03:40:52.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496868+0000", 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023520899999999998 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496903+0000", 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4443000000000002e-05 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=32, reqid=client.4615.0:302, at_version=49'23, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.491000+0000", 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.712281425, 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065612769999999999, 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491000+0000", 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491000+0000", 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491001+0000", 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4600000000000003e-07 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492933+0000", 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019313749999999999 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492951+0000", 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8521000000000001e-05 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492960+0000", 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4089999999999992e-06 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494238+0000", 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012774049999999999 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494247+0000", 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0839999999999998e-06 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497522+0000", 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032754910000000002 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497562+0000", 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9045999999999999e-05 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 49/31 ECSubWrite(tid=36, reqid=client.4615.0:307, at_version=49'24, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.491223+0000", 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.712058247999998, 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057044000000000001, 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491223+0000", 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491223+0000", 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491224+0000", 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4399999999999998e-07 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491244+0000", 2024-09-16T03:40:52.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9286000000000001e-05 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491259+0000", 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5075999999999999e-05 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491267+0000", 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3930000000000008e-06 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495031+0000", 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037640339999999999 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495041+0000", 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9969999999999993e-06 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496907+0000", 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018654380000000001 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496928+0000", 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1231999999999999e-05 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 49/31 ECSubWrite(tid=83, reqid=osd.0.0:83, at_version=49'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.491639+0000", 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.711642965999999, 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047024830000000004, 2024-09-16T03:40:52.267 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491639+0000", 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491639+0000", 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491639+0000", 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6800000000000001e-07 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491666+0000", 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6568e-05 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491682+0000", 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6280999999999999e-05 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491691+0000", 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3729999999999999e-06 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491771+0000", 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0195e-05 2024-09-16T03:40:52.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491783+0000", 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1912e-05 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496306+0000", 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045223049999999999 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496341+0000", 2024-09-16T03:40:52.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5481000000000001e-05 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=33, reqid=client.4615.0:303, at_version=49'24, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.492978+0000", 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.710303240000002, 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046125319999999999, 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.272 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492978+0000", 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492978+0000", 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492979+0000", 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1600000000000001e-07 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492995+0000", 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5557999999999999e-05 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.493005+0000", 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0686000000000001e-05 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.493012+0000", 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0169999999999999e-06 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495579+0000", 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002566381 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495588+0000", 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4399999999999994e-06 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497566+0000", 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019781640000000001 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497591+0000", 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4669999999999999e-05 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=34, reqid=client.4615.0:303, at_version=0'0, trim_to=0'0, roll_forward_to=49'24))", 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.502228+0000", 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.701053824999999, 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005033989, 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502228+0000", 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502228+0000", 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502229+0000", 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5600000000000004e-07 2024-09-16T03:40:52.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502240+0000", 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1653e-05 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502273+0000", 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.286e-05 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502283+0000", 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5789999999999993e-06 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502366+0000", 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3085999999999997e-05 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502380+0000", 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.412e-05 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.507229+0000", 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048493770000000002 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.507262+0000", 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2458e-05 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 49/31 ECSubWrite(tid=32, reqid=osd.8.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=49'22))", 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.502355+0000", 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.700926696, 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048604859999999998, 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502355+0000", 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502355+0000", 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502356+0000", 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.95e-07 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502368+0000", 2024-09-16T03:40:52.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1558e-05 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502383+0000", 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5869999999999999e-05 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502392+0000", 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6680000000000007e-06 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502471+0000", 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8912999999999995e-05 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.502482+0000", 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0567000000000001e-05 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.507178+0000", 2024-09-16T03:40:52.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046967110000000001 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.507215+0000", 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7203999999999998e-05 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=35, reqid=client.4615.0:309, at_version=49'26, trim_to=0'0, roll_forward_to=49'24))", 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.504135+0000", 2024-09-16T03:40:52.286 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.699146988999999, 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054625289999999998, 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504135+0000", 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504135+0000", 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504136+0000", 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6400000000000001e-07 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504450+0000", 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031444699999999998 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504462+0000", 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2208999999999999e-05 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504471+0000", 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2129999999999997e-06 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504552+0000", 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0445000000000007e-05 2024-09-16T03:40:52.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.504562+0000", 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6819999999999998e-06 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.509553+0000", 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004991021 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.509597+0000", 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4647999999999997e-05 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 52/31 ECSubWrite(tid=44, reqid=client.4615.0:337, at_version=52'36, trim_to=0'0, roll_forward_to=51'34))", 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:31.494440+0000", 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.708841691, 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043633559999999997, 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.291 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.494440+0000", 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.494440+0000", 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.494440+0000", 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6e-07 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.495052+0000", 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061194699999999995 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.495065+0000", 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3223e-05 2024-09-16T03:40:52.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.495073+0000", 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6550000000000004e-06 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.495155+0000", 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2068000000000001e-05 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.495162+0000", 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4510000000000003e-06 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.498762+0000", 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035996579999999999 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:31.498803+0000", 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1193999999999999e-05 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.295 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 54/31 ECSubWrite(tid=56, reqid=osd.6.0:56, at_version=54'45, trim_to=0'0, roll_forward_to=53'43))", 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:33.510572+0000", 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.692709849, 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043840609999999999, 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:33.510572+0000", 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:33.510572+0000", 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:33.510572+0000", 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3899999999999999e-07 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:33.510594+0000", 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2124999999999999e-05 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:33.510601+0000", 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2060000000000001e-06 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:33.510610+0000", 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.064e-06 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:33.510676+0000", 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5430000000000002e-05 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:33.510687+0000", 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1245e-05 2024-09-16T03:40:52.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:33.514915+0000", 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042284219999999999 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:33.514956+0000", 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0429999999999997e-05 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/31 ECSubWrite(tid=47, reqid=client.4615.0:467, at_version=59'36, trim_to=0'0, roll_forward_to=55'34))", 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:38.491189+0000", 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.712092243000001, 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005777936, 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:38.491189+0000", 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:38.491189+0000", 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:38.491190+0000", 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0999999999999999e-07 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:38.492878+0000", 2024-09-16T03:40:52.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016884319999999999 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:38.492890+0000", 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2097e-05 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:38.492898+0000", 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0639999999999994e-06 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:38.492955+0000", 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6876999999999998e-05 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:38.492963+0000", 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7849999999999992e-06 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:38.496937+0000", 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039740510000000001 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:38.496967+0000", 2024-09-16T03:40:52.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0219999999999999e-05 2024-09-16T03:40:52.305 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.305 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.305 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.305 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.305 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.305 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:52.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:40:52.307 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:52.305+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:52.407 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:52.406+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:52.411 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:52.411 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:52.411 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:52.412 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:52.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:40:52.477 INFO:tasks.rados.rados.0.smithi121.stdout:245: done (4 left) 2024-09-16T03:40:52.477 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 48 current snap is 24 2024-09-16T03:40:52.477 INFO:tasks.rados.rados.0.smithi121.stdout:246: seq_num 117 ranges {802816=425984,1228800=606208,1835008=540672,2375680=409600} 2024-09-16T03:40:52.483 INFO:tasks.rados.rados.0.smithi121.stdout:246: writing smithi12138290-48 from 802816 to 1228800 tid 1 2024-09-16T03:40:52.487 INFO:tasks.rados.rados.0.smithi121.stdout:246: writing smithi12138290-48 from 1228800 to 1835008 tid 2 2024-09-16T03:40:52.494 INFO:tasks.rados.rados.0.smithi121.stdout:246: writing smithi12138290-48 from 1835008 to 2375680 tid 3 2024-09-16T03:40:52.495 INFO:tasks.rados.rados.0.smithi121.stdout:246: writing smithi12138290-48 from 2375680 to 2785280 tid 4 2024-09-16T03:40:52.496 INFO:tasks.rados.rados.0.smithi121.stdout:247: setattr oid 7 current snap is 24 2024-09-16T03:40:52.496 INFO:tasks.rados.rados.0.smithi121.stdout:241: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-16T03:40:52.508 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:52.506+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:52.520 INFO:tasks.rados.rados.0.smithi121.stdout:242: finishing write tid 1 to smithi12138290-1 2024-09-16T03:40:52.520 INFO:tasks.rados.rados.0.smithi121.stdout:242: finishing write tid 2 to smithi12138290-1 2024-09-16T03:40:52.520 INFO:tasks.rados.rados.0.smithi121.stdout:242: finishing write tid 3 to smithi12138290-1 2024-09-16T03:40:52.520 INFO:tasks.rados.rados.0.smithi121.stdout:242: finishing write tid 4 to smithi12138290-1 2024-09-16T03:40:52.520 INFO:tasks.rados.rados.0.smithi121.stdout:242: finishing write tid 5 to smithi12138290-1 2024-09-16T03:40:52.520 INFO:tasks.rados.rados.0.smithi121.stdout:244: finishing rollback tid 0 to smithi12138290-26 2024-09-16T03:40:52.520 INFO:tasks.rados.rados.0.smithi121.stdout:243: finishing rollback tid 0 to smithi12138290-34 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:242: finishing write tid 6 to smithi12138290-1 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:242: oid 1 updating version 0 to 82 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:242: oid 1 version 82 is already newer than 81 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:242: oid 1 updating version 82 to 83 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:242: oid 1 updating version 83 to 85 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:242: oid 1 version 85 is already newer than 84 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 1 v 85 (ObjNum 116 snap 24 seq_num 116) dirty exists 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:242: left oid 1 (ObjNum 116 snap 24 seq_num 116) 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:244: finishing rollback tid 1 to smithi12138290-26 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:244: finishing rollback tid 2 to smithi12138290-26 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 26 v 78 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:243: finishing rollback tid 1 to smithi12138290-34 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:243: finishing rollback tid 2 to smithi12138290-34 2024-09-16T03:40:52.521 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 34 v 37 (ObjNum 103 snap 20 seq_num 103) dirty exists 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: finishing write tid 1 to smithi12138290-48 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: finishing write tid 2 to smithi12138290-48 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: finishing write tid 3 to smithi12138290-48 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: finishing write tid 4 to smithi12138290-48 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: finishing write tid 5 to smithi12138290-48 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: finishing write tid 6 to smithi12138290-48 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: oid 48 updating version 0 to 47 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: oid 48 version 47 is already newer than 46 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: oid 48 updating version 47 to 48 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: oid 48 version 48 is already newer than 45 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: oid 48 version 48 is already newer than 44 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 48 v 48 (ObjNum 117 snap 24 seq_num 117) dirty exists 2024-09-16T03:40:52.522 INFO:tasks.rados.rados.0.smithi121.stdout:246: left oid 48 (ObjNum 117 snap 24 seq_num 117) 2024-09-16T03:40:52.523 INFO:tasks.rados.rados.0.smithi121.stdout:241: done (5 left) 2024-09-16T03:40:52.523 INFO:tasks.rados.rados.0.smithi121.stdout:242: done (4 left) 2024-09-16T03:40:52.523 INFO:tasks.rados.rados.0.smithi121.stdout:243: done (3 left) 2024-09-16T03:40:52.523 INFO:tasks.rados.rados.0.smithi121.stdout:244: done (2 left) 2024-09-16T03:40:52.523 INFO:tasks.rados.rados.0.smithi121.stdout:246: done (1 left) 2024-09-16T03:40:52.523 INFO:tasks.rados.rados.0.smithi121.stdout:248: read oid 27 snap 15 2024-09-16T03:40:52.523 INFO:tasks.rados.rados.0.smithi121.stdout:248: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:40:52.523 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 17 current snap is 24 2024-09-16T03:40:52.523 INFO:tasks.rados.rados.0.smithi121.stdout:249: seq_num 118 ranges {1998848=442368,2441216=753664,3194880=655360,3850240=147456} 2024-09-16T03:40:52.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:40:52.534 INFO:tasks.rados.rados.0.smithi121.stdout:249: writing smithi12138290-17 from 1998848 to 2441216 tid 1 2024-09-16T03:40:52.539 INFO:tasks.rados.rados.0.smithi121.stdout:249: writing smithi12138290-17 from 2441216 to 3194880 tid 2 2024-09-16T03:40:52.544 INFO:tasks.rados.rados.0.smithi121.stdout:249: writing smithi12138290-17 from 3194880 to 3850240 tid 3 2024-09-16T03:40:52.545 INFO:tasks.rados.rados.0.smithi121.stdout:249: writing smithi12138290-17 from 3850240 to 3997696 tid 4 2024-09-16T03:40:52.546 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 20 current snap is 24 2024-09-16T03:40:52.546 INFO:tasks.rados.rados.0.smithi121.stdout:250: seq_num 119 ranges {1409024=458752,1867776=507904,2375680=507904,2883584=458752,3342336=81920} 2024-09-16T03:40:52.556 INFO:tasks.rados.rados.0.smithi121.stdout:250: writing smithi12138290-20 from 1409024 to 1867776 tid 1 2024-09-16T03:40:52.560 INFO:tasks.rados.rados.0.smithi121.stdout:250: writing smithi12138290-20 from 1867776 to 2375680 tid 2 2024-09-16T03:40:52.563 INFO:tasks.rados.rados.0.smithi121.stdout:250: writing smithi12138290-20 from 2375680 to 2883584 tid 3 2024-09-16T03:40:52.565 INFO:tasks.rados.rados.0.smithi121.stdout:250: writing smithi12138290-20 from 2883584 to 3342336 tid 4 2024-09-16T03:40:52.566 INFO:tasks.rados.rados.0.smithi121.stdout:250: writing smithi12138290-20 from 3342336 to 3424256 tid 5 2024-09-16T03:40:52.566 INFO:tasks.rados.rados.0.smithi121.stdout:251: rollback oid 29 current snap is 24 2024-09-16T03:40:52.566 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 29 to 22 2024-09-16T03:40:52.566 INFO:tasks.rados.rados.0.smithi121.stdout:248: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:40:52.571 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 7 v 81 (ObjNum 89 snap 12 seq_num 89) dirty exists 2024-09-16T03:40:52.571 INFO:tasks.rados.rados.0.smithi121.stdout:249: finishing write tid 1 to smithi12138290-17 2024-09-16T03:40:52.571 INFO:tasks.rados.rados.0.smithi121.stdout:249: finishing write tid 2 to smithi12138290-17 2024-09-16T03:40:52.571 INFO:tasks.rados.rados.0.smithi121.stdout:249: finishing write tid 3 to smithi12138290-17 2024-09-16T03:40:52.571 INFO:tasks.rados.rados.0.smithi121.stdout:249: finishing write tid 4 to smithi12138290-17 2024-09-16T03:40:52.571 INFO:tasks.rados.rados.0.smithi121.stdout:247: done (4 left) 2024-09-16T03:40:52.571 INFO:tasks.rados.rados.0.smithi121.stdout:248: done (3 left) 2024-09-16T03:40:52.571 INFO:tasks.rados.rados.0.smithi121.stdout:252: rmattr oid 34 current snap is 24 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:249: finishing write tid 5 to smithi12138290-17 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:249: finishing write tid 6 to smithi12138290-17 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:249: oid 17 updating version 0 to 94 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:249: oid 17 version 94 is already newer than 92 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:249: oid 17 updating version 94 to 95 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:249: oid 17 updating version 95 to 96 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:249: oid 17 version 96 is already newer than 93 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 17 v 96 (ObjNum 118 snap 24 seq_num 118) dirty exists 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:249: left oid 17 (ObjNum 118 snap 24 seq_num 118) 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:250: finishing write tid 1 to smithi12138290-20 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:249: done (3 left) 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:253: copy_from oid 19 from oid 38 current snap is 24 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:250: finishing write tid 2 to smithi12138290-20 2024-09-16T03:40:52.572 INFO:tasks.rados.rados.0.smithi121.stdout:251: finishing rollback tid 0 to smithi12138290-29 2024-09-16T03:40:52.573 INFO:tasks.rados.rados.0.smithi121.stdout:250: finishing write tid 3 to smithi12138290-20 2024-09-16T03:40:52.573 INFO:tasks.rados.rados.0.smithi121.stdout:254: snap_remove snap 17 2024-09-16T03:40:52.593 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:52.593 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:52.593 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:52.594 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:52.594 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:52.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:40:52.609 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:52.607+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:52.709 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:52.708+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:52.748 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:52.748 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:52.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:52.748 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:52.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4341.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.484472351999997, 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59293511399999999, 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4341", 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:40:52.749 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260707+0000", 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.66e-06 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260713+0000", 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6150000000000002e-06 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260728+0000", 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5333e-05 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260741+0000", 2024-09-16T03:40:52.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-09-16T03:40:52.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260825+0000", 2024-09-16T03:40:52.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4147999999999995e-05 2024-09-16T03:40:52.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260836+0000", 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0351e-05 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852485+0000", 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59164876099999997 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852573+0000", 2024-09-16T03:40:52.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8352999999999999e-05 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852639+0000", 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853142+0000", 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050253099999999996 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853585+0000", 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000443666 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853615+0000", 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9774e-05 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853640+0000", 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4924999999999999e-05 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.755 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.028063535000001, 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093665129999999999, 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.757 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-09-16T03:40:52.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.757 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:52.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717114+0000", 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.06e-07 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717170+0000", 2024-09-16T03:40:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5937000000000003e-05 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717182+0000", 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1844999999999999e-05 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717188+0000", 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.054e-06 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.724763+0000", 2024-09-16T03:40:52.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075747119999999999 2024-09-16T03:40:52.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.724809+0000", 2024-09-16T03:40:52.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5933999999999998e-05 2024-09-16T03:40:52.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.724882+0000", 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3650000000000001e-05 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726043+0000", 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011606260000000001 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726385+0000", 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034193800000000002 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726443+0000", 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8251999999999997e-05 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726480+0000", 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6959000000000003e-05 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.027974294000003, 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095750269999999998, 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.763 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717203+0000", 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7800000000000001e-07 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717208+0000", 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.109e-06 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717215+0000", 2024-09-16T03:40:52.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0210000000000002e-06 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717226+0000", 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0934000000000001e-05 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725719+0000", 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084932659999999993 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725774+0000", 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5260000000000003e-05 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.725850+0000", 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5907000000000004e-05 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726495+0000", 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064497100000000004 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726716+0000", 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022091799999999999 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726744+0000", 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8561e-05 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726778+0000", 2024-09-16T03:40:52.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3501999999999999e-05 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.717235+0000", 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.027942095999997, 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097518989999999996, 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.769 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717235+0000", 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717235+0000", 2024-09-16T03:40:52.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717236+0000", 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8400000000000005e-07 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717240+0000", 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717247+0000", 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4939999999999997e-06 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.717252+0000", 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726072+0000", 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088197359999999999 2024-09-16T03:40:52.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726119+0000", 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6662000000000001e-05 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726196+0000", 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6958000000000004e-05 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726593+0000", 2024-09-16T03:40:52.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000397272 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726935+0000", 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034162800000000002 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726964+0000", 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9268000000000002e-05 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.726987+0000", 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3096e-05 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.593908352, 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094025229999999994, 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.775 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 162 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151269+0000", 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151271+0000", 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2510000000000001e-06 2024-09-16T03:40:52.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151275+0000", 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4199999999999999e-06 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151278+0000", 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8669999999999998e-06 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.156802+0000", 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055245119999999997 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.156852+0000", 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0195999999999998e-05 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.156927+0000", 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4192999999999999e-05 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159094+0000", 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021670740000000002 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160511+0000", 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001417583 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160599+0000", 2024-09-16T03:40:52.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7965000000000002e-05 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160671+0000", 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2113999999999994e-05 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.593895484000001, 2024-09-16T03:40:52.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094948329999999994, 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 163 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151282+0000", 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.11e-07 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151284+0000", 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2680000000000001e-06 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151287+0000", 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2820000000000001e-06 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151291+0000", 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2100000000000002e-06 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157351+0000", 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060598270000000003 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157437+0000", 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6279999999999994e-05 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157555+0000", 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118377 2024-09-16T03:40:52.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159098+0000", 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015432499999999999 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160709+0000", 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016109799999999999 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160743+0000", 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3432000000000001e-05 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.160777+0000", 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3716e-05 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151306+0000", 2024-09-16T03:40:52.787 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.593870764000002, 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01219754, 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151306+0000", 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151306+0000", 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151307+0000", 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4999999999999999e-07 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151364+0000", 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7055000000000002e-05 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151371+0000", 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0550000000000002e-06 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151374+0000", 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.534e-06 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157903+0000", 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065283620000000002 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.157969+0000", 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6211999999999996e-05 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158097+0000", 2024-09-16T03:40:52.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127737 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159104+0000", 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010076919999999999 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163354+0000", 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042495579999999996 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163439+0000", 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5276000000000001e-05 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163504+0000", 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4808999999999996e-05 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.793 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.593796425000001, 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012275085, 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 165 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151381+0000", 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3799999999999998e-07 2024-09-16T03:40:52.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151384+0000", 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5639999999999999e-06 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151388+0000", 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9280000000000003e-06 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.151391+0000", 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0699999999999998e-06 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158639+0000", 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072479620000000002 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158736+0000", 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7435000000000005e-05 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.158872+0000", 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000136119 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.159390+0000", 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051745999999999997 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163597+0000", 2024-09-16T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042074870000000002 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163631+0000", 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4057000000000003e-05 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.163656+0000", 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4664999999999998e-05 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:183 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283197+0000", 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.461979835999998, 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093764810000000007, 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 183 2024-09-16T03:40:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283197+0000", 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283197+0000", 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283198+0000", 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1199999999999999e-07 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283200+0000", 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1780000000000002e-06 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283203+0000", 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.41e-06 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283206+0000", 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5730000000000002e-06 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286007+0000", 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028007969999999998 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286078+0000", 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1606000000000005e-05 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286188+0000", 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000109978 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288028+0000", 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018400619999999999 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292533+0000", 2024-09-16T03:40:52.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045046419999999997 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292556+0000", 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2733000000000001e-05 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292574+0000", 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.819e-05 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:184 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.461968011000003, 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094925180000000001, 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.806 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 184 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283209+0000", 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0699999999999999e-07 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283211+0000", 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8810000000000001e-06 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283214+0000", 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1700000000000001e-06 2024-09-16T03:40:52.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283219+0000", 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1989999999999999e-06 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286487+0000", 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032681239999999999 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286562+0000", 2024-09-16T03:40:52.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5328999999999997e-05 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.286677+0000", 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114856 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289031+0000", 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023536310000000001 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292619+0000", 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035884580000000001 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292667+0000", 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8282000000000003e-05 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292702+0000", 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4381000000000002e-05 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.461949187000002, 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095539089999999993, 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.812 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 185 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283228+0000", 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4600000000000001e-07 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283260+0000", 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1488000000000001e-05 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283264+0000", 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-09-16T03:40:52.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283267+0000", 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0299999999999998e-06 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287022+0000", 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037554400000000001 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287090+0000", 2024-09-16T03:40:52.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7329999999999994e-05 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287210+0000", 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119838 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289036+0000", 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001826814 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292735+0000", 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036988400000000001 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292762+0000", 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7036e-05 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292782+0000", 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9690999999999998e-05 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.817 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283295+0000", 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.461882035000002, 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0096111779999999997, 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.818 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 186 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283295+0000", 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283295+0000", 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283296+0000", 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0299999999999999e-07 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283299+0000", 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6500000000000002e-06 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283304+0000", 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8790000000000002e-06 2024-09-16T03:40:52.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283307+0000", 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.061e-06 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287556+0000", 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042482650000000002 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287625+0000", 2024-09-16T03:40:52.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8989e-05 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.287736+0000", 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111983 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289043+0000", 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013063020000000001 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292823+0000", 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037805619999999999 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292871+0000", 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7482999999999999e-05 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292906+0000", 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5500999999999998e-05 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.461864581, 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0096710279999999999, 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.824 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283313+0000", 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283315+0000", 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.227e-06 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283322+0000", 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2119999999999997e-06 2024-09-16T03:40:52.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283325+0000", 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9000000000000002e-06 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288058+0000", 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047327790000000003 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288104+0000", 2024-09-16T03:40:52.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5788999999999998e-05 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288174+0000", 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9769000000000002e-05 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289048+0000", 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087418399999999996 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292940+0000", 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038922129999999998 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292965+0000", 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4929000000000001e-05 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.292984+0000", 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8695999999999999e-05 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283335+0000", 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.461841862, 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097754880000000006, 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 188 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283335+0000", 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283335+0000", 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283336+0000", 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8599999999999999e-07 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283372+0000", 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6279000000000003e-05 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283376+0000", 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.454e-06 2024-09-16T03:40:52.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283379+0000", 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288401+0000", 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050215930000000004 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288476+0000", 2024-09-16T03:40:52.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4667999999999997e-05 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.288596+0000", 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120439 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289367+0000", 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00077095699999999996 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293025+0000", 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036582390000000002 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293075+0000", 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0050999999999999e-05 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293111+0000", 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5525999999999998e-05 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.835 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4257.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:59.283384+0000", 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.461792932999998, 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097953789999999999, 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4257", 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/2518569325", 2024-09-16T03:40:52.836 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 189 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283384+0000", 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283384+0000", 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283385+0000", 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7799999999999997e-07 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283387+0000", 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2330000000000001e-06 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283391+0000", 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.878e-06 2024-09-16T03:40:52.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.283393+0000", 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.525e-06 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289066+0000", 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056729689999999999 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289107+0000", 2024-09-16T03:40:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0513999999999999e-05 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289176+0000", 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9481999999999996e-05 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.289693+0000", 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051673699999999997 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293139+0000", 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034465229999999999 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293163+0000", 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3811999999999998e-05 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:59.293180+0000", 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6427999999999999e-05 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:483 3.8s0 3:1676b3ea:::smithi12138290-33:head [copy-from ver 28] snapc 13=[13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:42.473301+0000", 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.271876364000001, 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021928366000000001, 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:52.842 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 483 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473301+0000", 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473301+0000", 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473303+0000", 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1009999999999999e-06 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473315+0000", 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473363+0000", 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7976000000000003e-05 2024-09-16T03:40:52.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473371+0000", 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8869999999999998e-06 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473437+0000", 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5574999999999995e-05 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473444+0000", 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6089999999999997e-06 2024-09-16T03:40:52.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.475559+0000", 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021149239999999998 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.475702+0000", 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014371699999999999 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.492468+0000", 2024-09-16T03:40:52.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016765841 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.494270+0000", 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018016639999999999 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.495153+0000", 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000883434 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.495229+0000", 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5977999999999997e-05 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:484 3.8s0 3:1676b3ea:::smithi12138290-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:42.473483+0000", 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.271694083, 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020673568, 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.848 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 484 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473483+0000", 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473483+0000", 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473484+0000", 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.850000000000001e-07 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473489+0000", 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9409999999999998e-06 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473498+0000", 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9339999999999995e-06 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473505+0000", 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9070000000000001e-06 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473736+0000", 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000231179 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473743+0000", 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.861e-06 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.476480+0000", 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027373520000000002 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.476532+0000", 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2021000000000002e-05 2024-09-16T03:40:52.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.493932+0000", 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017399465999999999 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.494046+0000", 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114605 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.494157+0000", 2024-09-16T03:40:52.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011031699999999999 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs3 73/55 ECSubWrite(tid=174, reqid=osd.0.0:174, at_version=73'30, trim_to=0'0, roll_forward_to=56'28))", 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.478875+0000", 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.26630201599999997, 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093149649999999997, 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478875+0000", 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478875+0000", 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478875+0000", 2024-09-16T03:40:52.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6999999999999999e-07 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478893+0000", 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7447000000000001e-05 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478903+0000", 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8919999999999995e-06 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478910+0000", 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7800000000000001e-06 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479220+0000", 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030927800000000002 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479228+0000", 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4030000000000007e-06 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488153+0000", 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089249220000000001 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488190+0000", 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7073000000000002e-05 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.es1 73/55 ECSubWrite(tid=24, reqid=osd.3.0:24, at_version=73'39, trim_to=0'0, roll_forward_to=72'35))", 2024-09-16T03:40:52.858 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.478929+0000", 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.26624789999999998, 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010026386, 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478929+0000", 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478929+0000", 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478930+0000", 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.15e-07 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478942+0000", 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2496e-05 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478959+0000", 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6464e-05 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.478966+0000", 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8349999999999998e-06 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479055+0000", 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9049999999999996e-05 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479078+0000", 2024-09-16T03:40:52.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3530999999999999e-05 2024-09-16T03:40:52.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488925+0000", 2024-09-16T03:40:52.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098466270000000002 2024-09-16T03:40:52.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488956+0000", 2024-09-16T03:40:52.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0967999999999999e-05 2024-09-16T03:40:52.864 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.864 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s4 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.484457+0000", 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.26072065300000002, 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098936110000000001, 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:52.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484457+0000", 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484457+0000", 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484457+0000", 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484472+0000", 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5500000000000001e-05 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484483+0000", 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0393e-05 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484489+0000", 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.348e-06 2024-09-16T03:40:52.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485669+0000", 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011799790000000001 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485679+0000", 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9690000000000006e-06 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494327+0000", 2024-09-16T03:40:52.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086475719999999992 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494350+0000", 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3502e-05 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:52.869 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:52.870 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:52.868+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:52.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:40:52.930 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:40:52.971 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:52.970+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:40:53.016 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:53.016 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:53.016 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:53.016 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:53.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:40:53.029 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-09-16T03:40:53.029 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-09-16T03:40:53.029 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-09-16T03:40:53.029 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-09-16T03:40:53.029 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-09-16T03:40:53.029 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-09-16T03:40:53.030 INFO:teuthology.orchestra.run.smithi103.stderr:config diff 2024-09-16T03:40:53.030 INFO:teuthology.orchestra.run.smithi103.stderr:config diff get 2024-09-16T03:40:53.030 INFO:teuthology.orchestra.run.smithi103.stderr:config get 2024-09-16T03:40:53.030 INFO:teuthology.orchestra.run.smithi103.stderr:config help [] 2024-09-16T03:40:53.030 INFO:teuthology.orchestra.run.smithi103.stderr:config set ... 2024-09-16T03:40:53.030 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: invalid command 2024-09-16T03:40:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:40:53.031 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-13, ['dump_ops_in_flight'] 2024-09-16T03:40:53.073 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:53.071+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:53.125 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:53.126 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:53.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:40:53.180 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:53.179+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:53.242 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:53.242 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:53.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:53.242 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:53.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.242 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 42/31 ECSubWrite(tid=32, reqid=osd.14.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:53.242 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.563730+0000", 2024-09-16T03:40:53.242 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.675122071000001, 2024-09-16T03:40:53.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050799820000000002, 2024-09-16T03:40:53.243 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.243 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.243 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.244 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563730+0000", 2024-09-16T03:40:53.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563730+0000", 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563731+0000", 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1200000000000002e-07 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563741+0000", 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0399e-05 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563761+0000", 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9894999999999998e-05 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563771+0000", 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0237e-05 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563841+0000", 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9499000000000007e-05 2024-09-16T03:40:53.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563850+0000", 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2229999999999996e-06 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.568773+0000", 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004922605 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.568810+0000", 2024-09-16T03:40:53.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7412e-05 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:375 3.3s0 3:c98c73f3:::smithi12138290-40:head [append 0~671744] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.701562353, 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086953299999999994, 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:53.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:53.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.249 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 375 2024-09-16T03:40:53.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.249 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:40:53.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:40:53.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537291+0000", 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.59e-06 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537708+0000", 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041628699999999999 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537721+0000", 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3738e-05 2024-09-16T03:40:53.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537728+0000", 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4409999999999998e-06 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537798+0000", 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0543000000000005e-05 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:53.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537931+0000", 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013217700000000001 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.539173+0000", 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012421660000000001 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.540584+0000", 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001411673 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545889+0000", 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053041340000000003 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545985+0000", 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6581000000000002e-05 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:376 3.3s0 3:c98c73f3:::smithi12138290-40:head [append 0~770048] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.696846092000001, 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087274550000000003, 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 376 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542008+0000", 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.756e-06 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542428+0000", 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041977300000000002 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542440+0000", 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2894000000000001e-05 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542447+0000", 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.584e-06 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542503+0000", 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5483999999999999e-05 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542630+0000", 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012754500000000001 2024-09-16T03:40:53.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.543864+0000", 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012343090000000001 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545743+0000", 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018785799999999999 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:53.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550638+0000", 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048951790000000004 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550733+0000", 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5351000000000002e-05 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:377 3.3s0 3:c98c73f3:::smithi12138290-40:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:53.260 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:40:53.261 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.696390023999999, 2024-09-16T03:40:53.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084428420000000007, 2024-09-16T03:40:53.261 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.261 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.261 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:53.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:53.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:53.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.261 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 377 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542463+0000", 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2600000000000002e-07 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542467+0000", 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6040000000000003e-06 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542473+0000", 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6019999999999999e-06 2024-09-16T03:40:53.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542476+0000", 2024-09-16T03:40:53.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.393e-06 2024-09-16T03:40:53.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.544778+0000", 2024-09-16T03:40:53.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023019329999999999 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.544876+0000", 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7448000000000006e-05 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545337+0000", 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046103399999999998 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.547435+0000", 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020983320000000001 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550831+0000", 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033956920000000001 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550905+0000", 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4078000000000005e-05 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.267 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:378 3.3s0 3:c98c73f3:::smithi12138290-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:53.267 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:40:53.267 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.696353601999999, 2024-09-16T03:40:53.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011059941, 2024-09-16T03:40:53.267 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.267 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 378 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1699999999999998e-07 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542502+0000", 2024-09-16T03:40:53.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8040000000000002e-06 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542509+0000", 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3320000000000003e-06 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542515+0000", 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.49e-06 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.546046+0000", 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035316670000000001 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.546144+0000", 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8040000000000003e-05 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.551270+0000", 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005125597 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.551384+0000", 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011363599999999999 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.553558+0000", 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021747580000000002 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.75404548199999999, 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060036960000000002, 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:40:53.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:40:53.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0699999999999997e-07 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484830+0000", 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3292999999999999e-05 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484841+0000", 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0891e-05 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484848+0000", 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8070000000000002e-06 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484897+0000", 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8887999999999997e-05 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484904+0000", 2024-09-16T03:40:53.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8430000000000004e-06 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490773+0000", 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058694940000000003 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490810+0000", 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6973e-05 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=93, reqid=osd.10.0:93, at_version=73'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.75326214700000005, 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056680849999999998, 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5599999999999999e-07 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485609+0000", 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9238000000000001e-05 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485620+0000", 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0781000000000001e-05 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485630+0000", 2024-09-16T03:40:53.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7270000000000002e-06 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485684+0000", 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4517e-05 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485695+0000", 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0542e-05 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491225+0000", 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055304880000000001 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491258+0000", 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2635999999999997e-05 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=94, reqid=osd.10.0:94, at_version=73'42, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:40:53.283 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.75207436900000002, 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065321210000000001, 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0399999999999995e-07 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486805+0000", 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6245999999999999e-05 2024-09-16T03:40:53.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486815+0000", 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0239e-05 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486820+0000", 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3020000000000002e-06 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488642+0000", 2024-09-16T03:40:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001822001 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488651+0000", 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5639999999999997e-06 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493274+0000", 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046231290000000001 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493310+0000", 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5936000000000003e-05 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=95, reqid=client.4615.0:649, at_version=73'44, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.75035930699999998, 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097259600000000005, 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488494+0000", 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5099999999999998e-07 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488724+0000", 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000229764 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488736+0000", 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.221e-05 2024-09-16T03:40:53.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488744+0000", 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3110000000000006e-06 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492660+0000", 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039159759999999998 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492669+0000", 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3449999999999995e-06 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498200+0000", 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055302169999999996 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498219+0000", 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9185999999999999e-05 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.292 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s4 73/55 ECSubWrite(tid=96, reqid=osd.10.0:96, at_version=73'22, trim_to=0'0, roll_forward_to=56'20))", 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.492535+0000", 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.74631724899999996, 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052032290000000002, 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492535+0000", 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492535+0000", 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492536+0000", 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4900000000000005e-07 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492558+0000", 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2532999999999999e-05 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492570+0000", 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1953000000000001e-05 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492578+0000", 2024-09-16T03:40:53.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492634+0000", 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.643e-05 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492644+0000", 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5680000000000005e-06 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497703+0000", 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050589609999999998 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497738+0000", 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5117000000000003e-05 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=97, reqid=client.4615.0:650, at_version=73'45, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:40:53.297 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.74591040399999997, 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052967650000000002, 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0399999999999996e-07 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493161+0000", 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021878300000000001 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493171+0000", 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5929999999999995e-06 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493178+0000", 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2389999999999996e-06 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493367+0000", 2024-09-16T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000189581 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493389+0000", 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1389000000000001e-05 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498222+0000", 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048334980000000003 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498238+0000", 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6178e-05 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s4 73/55 ECSubWrite(tid=98, reqid=osd.10.0:98, at_version=73'24, trim_to=0'0, roll_forward_to=56'20))", 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.74518742199999999, 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0063600940000000002, 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.302 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493680+0000", 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4543e-05 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493688+0000", 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.687e-06 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493694+0000", 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8209999999999998e-06 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493766+0000", 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2169999999999995e-05 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493776+0000", 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2890000000000003e-06 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499989+0000", 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062135269999999999 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500025+0000", 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5596999999999998e-05 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=99, reqid=client.4615.0:651, at_version=73'46, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.494942+0000", 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.74391002399999995, 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052685420000000002, 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494942+0000", 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494942+0000", 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494943+0000", 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2200000000000003e-07 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495356+0000", 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041339099999999999 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495371+0000", 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4537e-05 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495379+0000", 2024-09-16T03:40:53.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6829999999999997e-06 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495467+0000", 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7392000000000006e-05 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495476+0000", 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5240000000000006e-06 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500158+0000", 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046818190000000003 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500211+0000", 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2473999999999999e-05 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=100, reqid=client.4615.0:652, at_version=73'47, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:40:53.311 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.74177977100000003, 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083283569999999998, 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497073+0000", 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.61e-07 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497223+0000", 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014987799999999999 2024-09-16T03:40:53.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497236+0000", 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3278999999999999e-05 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497245+0000", 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8130000000000002e-06 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497329+0000", 2024-09-16T03:40:53.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3671999999999994e-05 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497338+0000", 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7000000000000003e-06 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.505360+0000", 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080216100000000002 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.505401+0000", 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0744000000000002e-05 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=93, reqid=osd.14.0:92, at_version=0'0, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.73871463000000004, 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0079817680000000002, 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.316 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7400000000000002e-07 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500148+0000", 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037999999999999e-05 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500163+0000", 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5109999999999999e-05 2024-09-16T03:40:53.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500172+0000", 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8419999999999994e-06 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500224+0000", 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1759999999999999e-05 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500233+0000", 2024-09-16T03:40:53.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8359999999999998e-06 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508092+0000", 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078594029999999992 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508119+0000", 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7005e-05 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.320 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=101, reqid=client.4615.0:653, at_version=73'48, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.500387+0000", 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.738464911, 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077870200000000004, 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500387+0000", 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500387+0000", 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500388+0000", 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500412+0000", 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4386e-05 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500421+0000", 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1020000000000003e-06 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500433+0000", 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2011000000000001e-05 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500496+0000", 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500507+0000", 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0499000000000001e-05 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508155+0000", 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076484200000000004 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508174+0000", 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9011000000000001e-05 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=94, reqid=osd.14.0:94, at_version=73'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:40:53.325 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.73773031200000005, 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070195409999999998, 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6499999999999999e-07 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501143+0000", 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0565e-05 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501156+0000", 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3278999999999999e-05 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501164+0000", 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0900000000000005e-06 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501227+0000", 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2682999999999996e-05 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501245+0000", 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8108e-05 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508122+0000", 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068766180000000001 2024-09-16T03:40:53.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508141+0000", 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9533000000000001e-05 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=102, reqid=osd.10.0:102, at_version=73'50, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.73692112099999996, 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087707529999999992, 2024-09-16T03:40:53.330 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501932+0000", 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3e-07 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501949+0000", 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7711e-05 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501962+0000", 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2549e-05 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501969+0000", 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1720000000000001e-06 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502019+0000", 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0340000000000003e-05 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502028+0000", 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8570000000000001e-06 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510672+0000", 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086435399999999999 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510702+0000", 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9954000000000002e-05 2024-09-16T03:40:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s4 73/55 ECSubWrite(tid=103, reqid=osd.10.0:98, at_version=0'0, trim_to=0'0, roll_forward_to=73'24))", 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502008+0000", 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.73684384999999997, 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006105933, 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502008+0000", 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502008+0000", 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502009+0000", 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:40:53.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502016+0000", 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3660000000000003e-06 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502024+0000", 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.542e-06 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502030+0000", 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9379999999999997e-06 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502199+0000", 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016938200000000001 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502209+0000", 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508072+0000", 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058631170000000002 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508114+0000", 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2546999999999998e-05 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=95, reqid=osd.14.0:95, at_version=73'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.73676635099999999, 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085979139999999999, 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3099999999999998e-07 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502103+0000", 2024-09-16T03:40:53.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6769000000000002e-05 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502114+0000", 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1258e-05 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502121+0000", 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6429999999999998e-06 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502175+0000", 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3698e-05 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502183+0000", 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6589999999999996e-06 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510650+0000", 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084665680000000007 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:53.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:53.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:53.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510684+0000", 2024-09-16T03:40:53.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3887999999999998e-05 2024-09-16T03:40:53.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.344 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:53.344 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:53.344 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:53.345 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:53.343+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:53.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:40:53.447 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:53.444+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:53.450 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:53.450 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:53.451 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:53.451 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:53.458 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:40:53.521 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T03:40:53.529 INFO:tasks.rados.rados.0.smithi121.stdout:254: done (4 left) 2024-09-16T03:40:53.529 INFO:tasks.rados.rados.0.smithi121.stdout:255: rollback oid 41 current snap is 24 2024-09-16T03:40:53.529 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 41 to 9 2024-09-16T03:40:53.530 INFO:tasks.rados.rados.0.smithi121.stdout:253: finishing copy_from to smithi12138290-19 2024-09-16T03:40:53.530 INFO:tasks.rados.rados.0.smithi121.stdout:253: got expected ENOENT (src dne) 2024-09-16T03:40:53.530 INFO:tasks.rados.rados.0.smithi121.stdout:253: finishing copy_from racing read to smithi12138290-19 2024-09-16T03:40:53.530 INFO:tasks.rados.rados.0.smithi121.stdout:250: finishing write tid 4 to smithi12138290-20 2024-09-16T03:40:53.530 INFO:tasks.rados.rados.0.smithi121.stdout:251: finishing rollback tid 1 to smithi12138290-29 2024-09-16T03:40:53.530 INFO:tasks.rados.rados.0.smithi121.stdout:250: finishing write tid 5 to smithi12138290-20 2024-09-16T03:40:53.530 INFO:tasks.rados.rados.0.smithi121.stdout:251: finishing rollback tid 2 to smithi12138290-29 2024-09-16T03:40:53.530 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 29 v 60 (ObjNum 108 snap 22 seq_num 108) dirty exists 2024-09-16T03:40:53.530 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 34 v 42 (ObjNum 103 snap 20 seq_num 103) dirty exists 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:250: finishing write tid 6 to smithi12138290-20 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:250: finishing write tid 7 to smithi12138290-20 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:250: oid 20 updating version 0 to 48 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:250: oid 20 updating version 48 to 49 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:250: oid 20 updating version 49 to 50 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:250: oid 20 version 50 is already newer than 47 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:250: oid 20 updating version 50 to 51 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:250: oid 20 version 51 is already newer than 46 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 20 v 51 (ObjNum 119 snap 24 seq_num 119) dirty exists 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:250: left oid 20 (ObjNum 119 snap 24 seq_num 119) 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:250: done (4 left) 2024-09-16T03:40:53.531 INFO:tasks.rados.rados.0.smithi121.stdout:251: done (3 left) 2024-09-16T03:40:53.532 INFO:tasks.rados.rados.0.smithi121.stdout:252: done (2 left) 2024-09-16T03:40:53.532 INFO:tasks.rados.rados.0.smithi121.stdout:253: done (1 left) 2024-09-16T03:40:53.532 INFO:tasks.rados.rados.0.smithi121.stdout:256: rmattr oid 36 current snap is 24 2024-09-16T03:40:53.532 INFO:tasks.rados.rados.0.smithi121.stdout:256: done (1 left) 2024-09-16T03:40:53.532 INFO:tasks.rados.rados.0.smithi121.stdout:257: rollback oid 50 current snap is 24 2024-09-16T03:40:53.532 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 50 to 22 2024-09-16T03:40:53.532 INFO:tasks.rados.rados.0.smithi121.stdout:258: read oid 25 snap -1 2024-09-16T03:40:53.532 INFO:tasks.rados.rados.0.smithi121.stdout:258: expect (ObjNum 61 snap 2 seq_num 61) 2024-09-16T03:40:53.532 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 23 current snap is 24 2024-09-16T03:40:53.532 INFO:tasks.rados.rados.0.smithi121.stdout:259: seq_num 120 ranges {1409024=475136,1884160=655360,2539520=622592,3162112=278528} 2024-09-16T03:40:53.540 INFO:tasks.rados.rados.0.smithi121.stdout:259: writing smithi12138290-23 from 1409024 to 1884160 tid 1 2024-09-16T03:40:53.544 INFO:tasks.rados.rados.0.smithi121.stdout:259: writing smithi12138290-23 from 1884160 to 2539520 tid 2 2024-09-16T03:40:53.546 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:53.545+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:53.548 INFO:tasks.rados.rados.0.smithi121.stdout:259: writing smithi12138290-23 from 2539520 to 3162112 tid 3 2024-09-16T03:40:53.550 INFO:tasks.rados.rados.0.smithi121.stdout:259: writing smithi12138290-23 from 3162112 to 3440640 tid 4 2024-09-16T03:40:53.550 INFO:tasks.rados.rados.0.smithi121.stdout:260: snap_create 2024-09-16T03:40:53.551 INFO:tasks.rados.rados.0.smithi121.stdout:257: finishing rollback tid 0 to smithi12138290-50 2024-09-16T03:40:53.551 INFO:tasks.rados.rados.0.smithi121.stdout:257: finishing rollback tid 1 to smithi12138290-50 2024-09-16T03:40:53.551 INFO:tasks.rados.rados.0.smithi121.stdout:258: expect (ObjNum 61 snap 2 seq_num 61) 2024-09-16T03:40:53.576 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:53.577 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:53.577 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:40:53.577 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:40:53.577 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:53.585 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:40:53.592 INFO:tasks.rados.rados.0.smithi121.stdout:257: finishing rollback tid 2 to smithi12138290-50 2024-09-16T03:40:53.592 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 50 v 43 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:40:53.592 INFO:tasks.rados.rados.0.smithi121.stdout:255: finishing rollback tid 0 to smithi12138290-41 2024-09-16T03:40:53.592 INFO:tasks.rados.rados.0.smithi121.stdout:255: finishing rollback tid 1 to smithi12138290-41 2024-09-16T03:40:53.592 INFO:tasks.rados.rados.0.smithi121.stdout:255: finishing rollback tid 2 to smithi12138290-41 2024-09-16T03:40:53.593 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 41 v 64 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:40:53.648 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:53.646+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:53.733 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:53.733 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:40:53.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:40:53.733 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:40:53.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.733 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:226 3.fs0 3:fe9db9b8:::smithi12138290-24:head [append 0~376832] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:53.733 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:53.733 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.189682144000002, 2024-09-16T03:40:53.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095498700000000002, 2024-09-16T03:40:53.733 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 226 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540454+0000", 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540455+0000", 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540861+0000", 2024-09-16T03:40:53.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040549800000000001 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540874+0000", 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2974e-05 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540882+0000", 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540948+0000", 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6383999999999993e-05 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540954+0000", 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9730000000000002e-06 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543210+0000", 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022561209999999998 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543290+0000", 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9907999999999995e-05 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543922+0000", 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000631713 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545016+0000", 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001093896 2024-09-16T03:40:53.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.549959+0000", 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049431980000000002 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550004+0000", 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4858999999999999e-05 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:227 3.fs0 3:fe9db9b8:::smithi12138290-24:head [setxattr _header (56) in=7b] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.189228501999999, 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092805639999999998, 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 227 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.740 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:53.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540908+0000", 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540909+0000", 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6400000000000001e-07 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540912+0000", 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1439999999999999e-06 2024-09-16T03:40:53.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540919+0000", 2024-09-16T03:40:53.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4000000000000003e-06 2024-09-16T03:40:53.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540925+0000", 2024-09-16T03:40:53.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6069999999999998e-06 2024-09-16T03:40:53.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541006+0000", 2024-09-16T03:40:53.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1231999999999997e-05 2024-09-16T03:40:53.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:53.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541012+0000", 2024-09-16T03:40:53.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3609999999999997e-06 2024-09-16T03:40:53.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544371+0000", 2024-09-16T03:40:53.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033583739999999999 2024-09-16T03:40:53.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544424+0000", 2024-09-16T03:40:53.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3535000000000002e-05 2024-09-16T03:40:53.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544709+0000", 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028452099999999998 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.545291+0000", 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00058281499999999998 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550110+0000", 2024-09-16T03:40:53.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048181430000000004 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550188+0000", 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8668e-05 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:228 3.fs0 3:fe9db9b8:::smithi12138290-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.189187347999997, 2024-09-16T03:40:53.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011210417, 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 228 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:53.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:53.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:40:53.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0399999999999996e-07 2024-09-16T03:40:53.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540952+0000", 2024-09-16T03:40:53.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6860000000000002e-06 2024-09-16T03:40:53.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540960+0000", 2024-09-16T03:40:53.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9729999999999997e-06 2024-09-16T03:40:53.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540966+0000", 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7509999999999997e-06 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541041+0000", 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5216999999999995e-05 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541046+0000", 2024-09-16T03:40:53.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0000000000000004e-06 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544875+0000", 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038287339999999999 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544964+0000", 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9245000000000004e-05 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550540+0000", 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055763649999999998 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550618+0000", 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7908e-05 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552159+0000", 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015409340000000001 2024-09-16T03:40:53.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:345 3.fs0 3:fe9db9b8:::smithi12138290-24:head [copy-from ver 4] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:31.497360+0000", 2024-09-16T03:40:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.232776167000001, 2024-09-16T03:40:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013644073, 2024-09-16T03:40:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 345 2024-09-16T03:40:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497360+0000", 2024-09-16T03:40:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497360+0000", 2024-09-16T03:40:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497362+0000", 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.598e-06 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497371+0000", 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0180000000000007e-06 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497382+0000", 2024-09-16T03:40:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1562e-05 2024-09-16T03:40:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497389+0000", 2024-09-16T03:40:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4409999999999998e-06 2024-09-16T03:40:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497446+0000", 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7054e-05 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497451+0000", 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.339e-06 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499316+0000", 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001864517 2024-09-16T03:40:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499415+0000", 2024-09-16T03:40:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9077e-05 2024-09-16T03:40:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.505946+0000", 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006531294 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.507163+0000", 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001216649 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.510932+0000", 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037689820000000001 2024-09-16T03:40:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.511004+0000", 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2541999999999998e-05 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:346 3.fs0 3:fe9db9b8:::smithi12138290-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.232731656999999, 2024-09-16T03:40:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096185339999999998, 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 346 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0499999999999999e-07 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497407+0000", 2024-09-16T03:40:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497412+0000", 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3859999999999998e-06 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497415+0000", 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1439999999999999e-06 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497492+0000", 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6577999999999998e-05 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497497+0000", 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.023e-06 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499594+0000", 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020968380000000002 2024-09-16T03:40:53.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499632+0000", 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8133000000000002e-05 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.506768+0000", 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071363449999999997 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.506883+0000", 2024-09-16T03:40:53.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011502000000000001 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.507023+0000", 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013991399999999999 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds3 53/31 ECSubWrite(tid=27, reqid=client.4615.0:354, at_version=53'8, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.503762+0000", 2024-09-16T03:40:53.772 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.226374703000001, 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096166930000000008, 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503762+0000", 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503762+0000", 2024-09-16T03:40:53.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503762+0000", 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0500000000000004e-07 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503788+0000", 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5888999999999999e-05 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503798+0000", 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6849999999999996e-06 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503806+0000", 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1850000000000004e-06 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503878+0000", 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2565999999999996e-05 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503888+0000", 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7880000000000001e-06 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513344+0000", 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0094557970000000002 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513378+0000", 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4277999999999999e-05 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds3 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.505226+0000", 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.224909914000001, 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0081836519999999996, 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505226+0000", 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505226+0000", 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505227+0000", 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0699999999999999e-06 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505249+0000", 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1469000000000001e-05 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505261+0000", 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2383e-05 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505269+0000", 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8169999999999999e-06 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506841+0000", 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001571624 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506851+0000", 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8179999999999999e-06 2024-09-16T03:40:53.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513390+0000", 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065394889999999999 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513410+0000", 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9981999999999999e-05 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:382 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.542264+0000", 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.187872222999999, 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088709159999999995, 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 382 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.782 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542264+0000", 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542264+0000", 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542266+0000", 2024-09-16T03:40:53.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.519e-06 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542272+0000", 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0560000000000001e-06 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542283+0000", 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1265e-05 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542289+0000", 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9399999999999999e-06 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542345+0000", 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6131000000000002e-05 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542428+0000", 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3436999999999994e-05 2024-09-16T03:40:53.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.551135+0000", 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087065679999999996 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:383 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.187831741, 2024-09-16T03:40:53.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010993826999999999, 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 383 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542308+0000", 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6450000000000001e-06 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542313+0000", 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2970000000000003e-06 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542317+0000", 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5979999999999999e-06 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543899+0000", 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015818539999999999 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544026+0000", 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012688099999999999 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.553298+0000", 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092721520000000005 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.790 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:382 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.550355+0000", 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.179781330000001, 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090719059999999994, 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.791 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 382 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550355+0000", 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550355+0000", 2024-09-16T03:40:53.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550356+0000", 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3699999999999999e-07 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550361+0000", 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7759999999999997e-06 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550369+0000", 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3639999999999999e-06 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550376+0000", 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5440000000000001e-06 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.553482+0000", 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031052829999999999 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.553615+0000", 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013317099999999999 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.559427+0000", 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058119310000000002 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.795 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:383 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.179746434999998, 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010309801, 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 383 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.77e-07 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550392+0000", 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0490000000000002e-06 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550397+0000", 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9699999999999998e-06 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550401+0000", 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4039999999999999e-06 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.553954+0000", 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003552955 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554020+0000", 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6389000000000004e-05 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560700+0000", 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066796570000000003 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:384 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.179731145000002, 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011709803, 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 384 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7399999999999999e-07 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550408+0000", 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.526e-06 2024-09-16T03:40:53.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550412+0000", 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.408e-06 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550415+0000", 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8260000000000001e-06 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554733+0000", 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004318087 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554837+0000", 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010340699999999999 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.562115+0000", 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072782749999999998 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.804 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:392 3.cs0 3:38c596e1:::smithi12138290-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.552815+0000", 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.177321266, 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091352450000000002, 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.805 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 392 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552815+0000", 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552815+0000", 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552816+0000", 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9100000000000004e-07 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552818+0000", 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4679999999999999e-06 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552827+0000", 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8999999999999995e-06 2024-09-16T03:40:53.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.552834+0000", 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8870000000000003e-06 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.557200+0000", 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043658270000000001 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:53.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.557351+0000", 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000151722 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560770+0000", 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034184699999999998 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560830+0000", 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0352000000000001e-05 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.561950+0000", 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011200279999999999 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:469 3.cs0 3:38c596e1:::smithi12138290-26:head [copy-from ver 10] snapc 11=[11,10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:38.482894+0000", 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.247241767, 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0094610879999999994, 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.810 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 469 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482894+0000", 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482894+0000", 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482896+0000", 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8509999999999999e-06 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482903+0000", 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5479999999999998e-06 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482915+0000", 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2452999999999999e-05 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482924+0000", 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7390000000000005e-06 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482994+0000", 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0298999999999999e-05 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483098+0000", 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103646 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.487565+0000", 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044667689999999998 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.489179+0000", 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001614164 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492105+0000", 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029258959999999999 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492356+0000", 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000250723 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.816 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.816 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.8s5 73/31 ECSubWrite(tid=208, reqid=osd.11.0:208, at_version=73'21, trim_to=0'0, roll_forward_to=72'19))", 2024-09-16T03:40:53.816 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.480608+0000", 2024-09-16T03:40:53.816 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.2495280659999999, 2024-09-16T03:40:53.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092095489999999992, 2024-09-16T03:40:53.816 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.816 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.817 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480608+0000", 2024-09-16T03:40:53.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480608+0000", 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480609+0000", 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480631+0000", 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2042999999999999e-05 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480641+0000", 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9399999999999997e-06 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480648+0000", 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7789999999999996e-06 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480704+0000", 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5844999999999998e-05 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480714+0000", 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4800000000000007e-06 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489783+0000", 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090693849999999993 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489818+0000", 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4629999999999999e-05 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds3 73/31 ECSubWrite(tid=119, reqid=osd.12.0:119, at_version=73'56, trim_to=0'0, roll_forward_to=73'54))", 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.481031+0000", 2024-09-16T03:40:53.821 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.2491051, 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088613419999999995, 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481031+0000", 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481031+0000", 2024-09-16T03:40:53.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481032+0000", 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2799999999999995e-07 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481051+0000", 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9015e-05 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481065+0000", 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3719e-05 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481073+0000", 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3459999999999994e-06 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481127+0000", 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3761000000000002e-05 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481137+0000", 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0148e-05 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489714+0000", 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085767310000000006 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489892+0000", 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000178894 2024-09-16T03:40:53.825 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 73/31 ECSubWrite(tid=91, reqid=osd.14.0:91, at_version=73'73, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.482643+0000", 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.247492786, 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010037870000000001, 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482643+0000", 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482643+0000", 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482644+0000", 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.43e-07 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482667+0000", 2024-09-16T03:40:53.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3268000000000002e-05 2024-09-16T03:40:53.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482677+0000", 2024-09-16T03:40:53.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0488e-05 2024-09-16T03:40:53.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482686+0000", 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6999999999999997e-06 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482741+0000", 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4625e-05 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482746+0000", 2024-09-16T03:40:53.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6250000000000004e-06 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.492656+0000", 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099094620000000008 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.492681+0000", 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5559e-05 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.483373+0000", 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.2467635530000001, 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012019056, 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483373+0000", 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483373+0000", 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483373+0000", 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9900000000000001e-07 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483385+0000", 2024-09-16T03:40:53.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1656e-05 2024-09-16T03:40:53.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483392+0000", 2024-09-16T03:40:53.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2640000000000002e-06 2024-09-16T03:40:53.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483398+0000", 2024-09-16T03:40:53.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4160000000000001e-06 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487424+0000", 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040257720000000004 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487432+0000", 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2029999999999992e-06 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.495353+0000", 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079208460000000005 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.495392+0000", 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000001e-05 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s3 73/31 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=73'31, trim_to=0'0, roll_forward_to=66'29))", 2024-09-16T03:40:53.835 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.484471+0000", 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.245664917, 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082121030000000001, 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484471+0000", 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484471+0000", 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484472+0000", 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484497+0000", 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4905e-05 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484506+0000", 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5380000000000008e-06 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484515+0000", 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1479999999999995e-06 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484591+0000", 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5623000000000004e-05 2024-09-16T03:40:53.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484600+0000", 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0059999999999998e-06 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.492654+0000", 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080541480000000006 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.492683+0000", 2024-09-16T03:40:53.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9383e-05 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:664 3.cs0 3:32badfe7:::smithi12138290-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.544957+0000", 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.185178931, 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0081477240000000003, 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:40:53.840 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 664 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.544957+0000", 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.544957+0000", 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:40:53.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.544958+0000", 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.84e-07 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.544959+0000", 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.277e-06 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.544963+0000", 2024-09-16T03:40:53.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.544965+0000", 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7539999999999998e-06 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.548034+0000", 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030690600000000002 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.548081+0000", 2024-09-16T03:40:53.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6545000000000002e-05 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.552057+0000", 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003976009 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.552147+0000", 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0081999999999996e-05 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:40:53.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:40:53.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.553105+0000", 2024-09-16T03:40:53.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00095792399999999997 2024-09-16T03:40:53.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.845 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:40:53.845 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:40:53.845 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:53.846 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:53.847 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:53.774+0000 7f15a020c640 -1 osd.13 53 log_to_monitors true 2024-09-16T03:40:53.848 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:40:53.848 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:53.844+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:53.849 INFO:tasks.rados.rados.0.smithi121.stdout:259: finishing write tid 1 to smithi12138290-23 2024-09-16T03:40:53.849 INFO:tasks.rados.rados.0.smithi121.stdout:259: finishing write tid 2 to smithi12138290-23 2024-09-16T03:40:53.849 INFO:tasks.rados.rados.0.smithi121.stdout:259: finishing write tid 3 to smithi12138290-23 2024-09-16T03:40:53.849 INFO:tasks.rados.rados.0.smithi121.stdout:259: finishing write tid 4 to smithi12138290-23 2024-09-16T03:40:53.849 INFO:tasks.rados.rados.0.smithi121.stdout:259: finishing write tid 5 to smithi12138290-23 2024-09-16T03:40:53.849 INFO:tasks.rados.rados.0.smithi121.stdout:259: finishing write tid 6 to smithi12138290-23 2024-09-16T03:40:53.849 INFO:tasks.rados.rados.0.smithi121.stdout:259: oid 23 updating version 0 to 87 2024-09-16T03:40:53.849 INFO:tasks.rados.rados.0.smithi121.stdout:259: oid 23 version 87 is already newer than 85 2024-09-16T03:40:53.849 INFO:tasks.rados.rados.0.smithi121.stdout:259: oid 23 updating version 87 to 89 2024-09-16T03:40:53.849 INFO:tasks.rados.rados.0.smithi121.stdout:259: oid 23 version 89 is already newer than 86 2024-09-16T03:40:53.850 INFO:tasks.rados.rados.0.smithi121.stdout:259: oid 23 version 89 is already newer than 88 2024-09-16T03:40:53.850 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 23 v 89 (ObjNum 120 snap 24 seq_num 120) dirty exists 2024-09-16T03:40:53.850 INFO:tasks.rados.rados.0.smithi121.stdout:259: left oid 23 (ObjNum 120 snap 24 seq_num 120) 2024-09-16T03:40:53.947 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:53.946+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:53.996 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:40:53.996 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:40:53.996 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:40:53.996 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:40:54.006 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:40:54.011 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:54.011 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:54.029 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:54.030 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:54.046 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:54.047 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:54.048 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:54.045+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:54.063 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:54.063 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:54.081 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:54.098 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:54.112 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:54.112 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:54.115 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:54.115 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:54.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:40:54.132 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:54.132 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:54.149 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:54.149 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:54.149 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:54.147+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:54.165 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:54.166 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:54.182 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:54.183 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:54.199 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:54.199 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:54.215 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.13 2024-09-16T03:40:54.233 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:54.237 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:54.237 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:54.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:40:54.250 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:54.250 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:54.251 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:54.248+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:40:54.253 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:40:54.351 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:54.349+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:54.353 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:54.353 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:54.362 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:40:54.451 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:54.450+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:54.463 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:54.463 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:54.472 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:40:54.533 INFO:tasks.rados.rados.0.smithi121.stdout:255: done (4 left) 2024-09-16T03:40:54.533 INFO:tasks.rados.rados.0.smithi121.stdout:257: done (3 left) 2024-09-16T03:40:54.533 INFO:tasks.rados.rados.0.smithi121.stdout:258: done (2 left) 2024-09-16T03:40:54.533 INFO:tasks.rados.rados.0.smithi121.stdout:259: done (1 left) 2024-09-16T03:40:54.533 INFO:tasks.rados.rados.0.smithi121.stdout:260: done (0 left) 2024-09-16T03:40:54.533 INFO:tasks.rados.rados.0.smithi121.stdout:261: copy_from oid 14 from oid 30 current snap is 25 2024-09-16T03:40:54.533 INFO:tasks.rados.rados.0.smithi121.stdout:262: rollback oid 31 current snap is 25 2024-09-16T03:40:54.533 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 31 to 15 2024-09-16T03:40:54.533 INFO:tasks.rados.rados.0.smithi121.stdout:263: delete oid 47 current snap is 25 2024-09-16T03:40:54.537 INFO:tasks.rados.rados.0.smithi121.stdout:263: done (2 left) 2024-09-16T03:40:54.538 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 25 current snap is 25 2024-09-16T03:40:54.538 INFO:tasks.rados.rados.0.smithi121.stdout:264: seq_num 122 ranges {1409024=507904,1916928=425984,2342912=442368,2785280=540672,3325952=147456} 2024-09-16T03:40:54.548 INFO:tasks.rados.rados.0.smithi121.stdout:264: writing smithi12138290-25 from 1409024 to 1916928 tid 1 2024-09-16T03:40:54.551 INFO:tasks.rados.rados.0.smithi121.stdout:264: writing smithi12138290-25 from 1916928 to 2342912 tid 2 2024-09-16T03:40:54.552 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:54.550+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:54.554 INFO:tasks.rados.rados.0.smithi121.stdout:264: writing smithi12138290-25 from 2342912 to 2785280 tid 3 2024-09-16T03:40:54.557 INFO:tasks.rados.rados.0.smithi121.stdout:264: writing smithi12138290-25 from 2785280 to 3325952 tid 4 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:264: writing smithi12138290-25 from 3325952 to 3473408 tid 5 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:265: delete oid 11 current snap is 25 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:262: finishing rollback tid 0 to smithi12138290-31 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:262: finishing rollback tid 1 to smithi12138290-31 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:262: finishing rollback tid 2 to smithi12138290-31 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 31 v 28 (ObjNum 84 snap 12 seq_num 84) dirty exists 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:261: finishing copy_from racing read to smithi12138290-14 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:264: finishing write tid 1 to smithi12138290-25 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:264: finishing write tid 2 to smithi12138290-25 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:261: finishing copy_from to smithi12138290-14 2024-09-16T03:40:54.559 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 14 v 39 (ObjNum 81 snap 12 seq_num 81) dirty exists 2024-09-16T03:40:54.560 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:54.557+0000 7f1597935640 -1 osd.13 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:40:54.561 INFO:tasks.rados.rados.0.smithi121.stdout:264: finishing write tid 3 to smithi12138290-25 2024-09-16T03:40:54.562 INFO:tasks.rados.rados.0.smithi121.stdout:261: done (3 left) 2024-09-16T03:40:54.562 INFO:tasks.rados.rados.0.smithi121.stdout:262: done (2 left) 2024-09-16T03:40:54.562 INFO:tasks.rados.rados.0.smithi121.stdout:265: done (1 left) 2024-09-16T03:40:54.562 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 2 current snap is 25 2024-09-16T03:40:54.562 INFO:tasks.rados.rados.0.smithi121.stdout:266: seq_num 124 ranges {2768896=540672,3309568=720896,4030464=786432,4816896=49152} 2024-09-16T03:40:54.579 INFO:tasks.rados.rados.0.smithi121.stdout:266: writing smithi12138290-2 from 2768896 to 3309568 tid 1 2024-09-16T03:40:54.583 INFO:tasks.rados.rados.0.smithi121.stdout:266: writing smithi12138290-2 from 3309568 to 4030464 tid 2 2024-09-16T03:40:54.587 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:54.587 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:54.589 INFO:tasks.rados.rados.0.smithi121.stdout:266: writing smithi12138290-2 from 4030464 to 4816896 tid 3 2024-09-16T03:40:54.589 INFO:tasks.rados.rados.0.smithi121.stdout:266: writing smithi12138290-2 from 4816896 to 4866048 tid 4 2024-09-16T03:40:54.589 INFO:tasks.rados.rados.0.smithi121.stdout:267: read oid 13 snap 23 2024-09-16T03:40:54.589 INFO:tasks.rados.rados.0.smithi121.stdout:267: expect (ObjNum 113 snap 23 seq_num 113) 2024-09-16T03:40:54.589 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 9 current snap is 25 2024-09-16T03:40:54.589 INFO:tasks.rados.rados.0.smithi121.stdout:264: finishing write tid 4 to smithi12138290-25 2024-09-16T03:40:54.589 INFO:tasks.rados.rados.0.smithi121.stdout:264: finishing write tid 5 to smithi12138290-25 2024-09-16T03:40:54.589 INFO:tasks.rados.rados.0.smithi121.stdout:264: finishing write tid 6 to smithi12138290-25 2024-09-16T03:40:54.589 INFO:tasks.rados.rados.0.smithi121.stdout:268: seq_num 125 ranges {147456=557056,704512=475136,1179648=507904,1687552=475136,2162688=98304} 2024-09-16T03:40:54.592 INFO:tasks.rados.rados.0.smithi121.stdout:268: writing smithi12138290-9 from 147456 to 704512 tid 1 2024-09-16T03:40:54.594 INFO:tasks.rados.rados.0.smithi121.stdout:268: writing smithi12138290-9 from 704512 to 1179648 tid 2 2024-09-16T03:40:54.597 INFO:tasks.rados.rados.0.smithi121.stdout:268: writing smithi12138290-9 from 1179648 to 1687552 tid 3 2024-09-16T03:40:54.601 INFO:tasks.rados.rados.0.smithi121.stdout:268: writing smithi12138290-9 from 1687552 to 2162688 tid 4 2024-09-16T03:40:54.602 INFO:tasks.rados.rados.0.smithi121.stdout:268: writing smithi12138290-9 from 2162688 to 2260992 tid 5 2024-09-16T03:40:54.602 INFO:tasks.rados.rados.0.smithi121.stdout:269: read oid 3 snap 23 2024-09-16T03:40:54.602 INFO:tasks.rados.rados.0.smithi121.stdout:269: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-16T03:40:54.602 INFO:tasks.rados.rados.0.smithi121.stdout:270: snap_remove snap 22 2024-09-16T03:40:54.604 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:40:54.654 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:54.651+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:54.708 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:54.708 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:54.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:40:54.754 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:54.752+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:54.855 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:54.853+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:54.859 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:54.860 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:54.869 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:40:54.956 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:54.954+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:55.012 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:55.012 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:55.021 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:40:55.058 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:55.056+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:55.136 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:40:55.158 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:55.157+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:55.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:40:55.259 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:55.258+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:55.350 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:55.351 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:55.360 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:40:55.361 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:55.359+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:55.461 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:55.458+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:55.472 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:40:55.535 INFO:tasks.rados.rados.0.smithi121.stdout:270: done (5 left) 2024-09-16T03:40:55.535 INFO:tasks.rados.rados.0.smithi121.stdout:271: rollback oid 46 current snap is 25 2024-09-16T03:40:55.535 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 46 to 23 2024-09-16T03:40:55.535 INFO:tasks.rados.rados.0.smithi121.stdout:264: finishing write tid 7 to smithi12138290-25 2024-09-16T03:40:55.535 INFO:tasks.rados.rados.0.smithi121.stdout:264: oid 25 updating version 0 to 50 2024-09-16T03:40:55.535 INFO:tasks.rados.rados.0.smithi121.stdout:264: oid 25 updating version 50 to 51 2024-09-16T03:40:55.535 INFO:tasks.rados.rados.0.smithi121.stdout:264: oid 25 version 51 is already newer than 46 2024-09-16T03:40:55.535 INFO:tasks.rados.rados.0.smithi121.stdout:264: oid 25 version 51 is already newer than 48 2024-09-16T03:40:55.535 INFO:tasks.rados.rados.0.smithi121.stdout:264: oid 25 version 51 is already newer than 47 2024-09-16T03:40:55.535 INFO:tasks.rados.rados.0.smithi121.stdout:264: oid 25 version 51 is already newer than 49 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 25 v 51 (ObjNum 122 snap 25 seq_num 122) dirty exists 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:264: left oid 25 (ObjNum 122 snap 25 seq_num 122) 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: finishing write tid 1 to smithi12138290-2 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: finishing write tid 2 to smithi12138290-2 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: finishing write tid 3 to smithi12138290-2 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: finishing write tid 4 to smithi12138290-2 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: finishing write tid 5 to smithi12138290-2 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: finishing write tid 6 to smithi12138290-2 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: oid 2 updating version 0 to 107 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: oid 2 updating version 107 to 108 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: oid 2 version 108 is already newer than 106 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: oid 2 version 108 is already newer than 105 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:266: oid 2 version 108 is already newer than 104 2024-09-16T03:40:55.536 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 2 v 108 (ObjNum 124 snap 25 seq_num 124) dirty exists 2024-09-16T03:40:55.537 INFO:tasks.rados.rados.0.smithi121.stdout:266: left oid 2 (ObjNum 124 snap 25 seq_num 124) 2024-09-16T03:40:55.537 INFO:tasks.rados.rados.0.smithi121.stdout:268: finishing write tid 1 to smithi12138290-9 2024-09-16T03:40:55.537 INFO:tasks.rados.rados.0.smithi121.stdout:268: finishing write tid 2 to smithi12138290-9 2024-09-16T03:40:55.537 INFO:tasks.rados.rados.0.smithi121.stdout:268: finishing write tid 3 to smithi12138290-9 2024-09-16T03:40:55.537 INFO:tasks.rados.rados.0.smithi121.stdout:269: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-16T03:40:55.538 INFO:tasks.rados.rados.0.smithi121.stdout:268: finishing write tid 4 to smithi12138290-9 2024-09-16T03:40:55.538 INFO:tasks.rados.rados.0.smithi121.stdout:268: finishing write tid 5 to smithi12138290-9 2024-09-16T03:40:55.538 INFO:tasks.rados.rados.0.smithi121.stdout:267: expect (ObjNum 113 snap 23 seq_num 113) 2024-09-16T03:40:55.548 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:40:55.562 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:55.560+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:55.593 INFO:tasks.rados.rados.0.smithi121.stdout:268: finishing write tid 6 to smithi12138290-9 2024-09-16T03:40:55.593 INFO:tasks.rados.rados.0.smithi121.stdout:268: finishing write tid 7 to smithi12138290-9 2024-09-16T03:40:55.593 INFO:tasks.rados.rados.0.smithi121.stdout:268: oid 9 updating version 0 to 37 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:268: oid 9 version 37 is already newer than 36 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:268: oid 9 updating version 37 to 39 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:268: oid 9 version 39 is already newer than 38 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:268: oid 9 version 39 is already newer than 34 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:268: oid 9 version 39 is already newer than 35 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 9 v 39 (ObjNum 125 snap 25 seq_num 125) dirty exists 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:268: left oid 9 (ObjNum 125 snap 25 seq_num 125) 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:264: done (5 left) 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:266: done (4 left) 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:267: done (3 left) 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:268: done (2 left) 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:269: done (1 left) 2024-09-16T03:40:55.594 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 42 current snap is 25 2024-09-16T03:40:55.595 INFO:tasks.rados.rados.0.smithi121.stdout:271: finishing rollback tid 0 to smithi12138290-46 2024-09-16T03:40:55.595 INFO:tasks.rados.rados.0.smithi121.stdout:271: finishing rollback tid 1 to smithi12138290-46 2024-09-16T03:40:55.595 INFO:tasks.rados.rados.0.smithi121.stdout:272: seq_num 126 ranges {131072=573440,704512=622592,1327104=622592,1949696=311296} 2024-09-16T03:40:55.597 INFO:tasks.rados.rados.0.smithi121.stdout:272: writing smithi12138290-42 from 131072 to 704512 tid 1 2024-09-16T03:40:55.600 INFO:tasks.rados.rados.0.smithi121.stdout:272: writing smithi12138290-42 from 704512 to 1327104 tid 2 2024-09-16T03:40:55.604 INFO:tasks.rados.rados.0.smithi121.stdout:272: writing smithi12138290-42 from 1327104 to 1949696 tid 3 2024-09-16T03:40:55.606 INFO:tasks.rados.rados.0.smithi121.stdout:272: writing smithi12138290-42 from 1949696 to 2260992 tid 4 2024-09-16T03:40:55.606 INFO:tasks.rados.rados.0.smithi121.stdout:273: snap_create 2024-09-16T03:40:55.606 INFO:tasks.rados.rados.0.smithi121.stdout:271: finishing rollback tid 2 to smithi12138290-46 2024-09-16T03:40:55.606 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 46 v 51 (ObjNum 87 snap 12 seq_num 87) dirty exists 2024-09-16T03:40:55.606 INFO:tasks.rados.rados.0.smithi121.stdout:272: finishing write tid 1 to smithi12138290-42 2024-09-16T03:40:55.606 INFO:tasks.rados.rados.0.smithi121.stdout:272: finishing write tid 2 to smithi12138290-42 2024-09-16T03:40:55.609 INFO:tasks.rados.rados.0.smithi121.stdout:272: finishing write tid 3 to smithi12138290-42 2024-09-16T03:40:55.609 INFO:tasks.rados.rados.0.smithi121.stdout:272: finishing write tid 4 to smithi12138290-42 2024-09-16T03:40:55.609 INFO:tasks.rados.rados.0.smithi121.stdout:272: finishing write tid 5 to smithi12138290-42 2024-09-16T03:40:55.611 INFO:tasks.rados.rados.0.smithi121.stdout:272: finishing write tid 6 to smithi12138290-42 2024-09-16T03:40:55.611 INFO:tasks.rados.rados.0.smithi121.stdout:272: oid 42 updating version 0 to 51 2024-09-16T03:40:55.611 INFO:tasks.rados.rados.0.smithi121.stdout:272: oid 42 version 51 is already newer than 49 2024-09-16T03:40:55.611 INFO:tasks.rados.rados.0.smithi121.stdout:272: oid 42 version 51 is already newer than 50 2024-09-16T03:40:55.611 INFO:tasks.rados.rados.0.smithi121.stdout:272: oid 42 updating version 51 to 52 2024-09-16T03:40:55.611 INFO:tasks.rados.rados.0.smithi121.stdout:272: oid 42 version 52 is already newer than 48 2024-09-16T03:40:55.611 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 42 v 52 (ObjNum 126 snap 25 seq_num 126) dirty exists 2024-09-16T03:40:55.612 INFO:tasks.rados.rados.0.smithi121.stdout:272: left oid 42 (ObjNum 126 snap 25 seq_num 126) 2024-09-16T03:40:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:55.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:40:55.663 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:40:55.661+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:40:55.745 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:55.745 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:55.753 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:40:55.764 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:55.762+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:55.858 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:55.858 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:55.864 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:55.863+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:55.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:40:55.966 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:55.964+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:55.978 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:40:56.067 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:56.066+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:56.081 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:56.081 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:56.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:40:56.168 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:56.167+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:56.206 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:56.206 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:56.216 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:40:56.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:56.269 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:56.267+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:56.320 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:56.320 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:56.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:40:56.370 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:56.368+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:56.377 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:56.405 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:56.423 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:56.435 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:56.435 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:56.435 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:56.435 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:56.435 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.435 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=30, reqid=client.4615.0:314, at_version=49'20, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:56.435 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:56.435 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 27.938795130999999, 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0049994970000000003, 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.436 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493571+0000", 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6100000000000002e-07 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.437 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493601+0000", 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9561999999999999e-05 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493614+0000", 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3421000000000001e-05 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.438 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493623+0000", 2024-09-16T03:40:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4050000000000006e-06 2024-09-16T03:40:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493705+0000", 2024-09-16T03:40:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1500000000000002e-05 2024-09-16T03:40:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493719+0000", 2024-09-16T03:40:56.440 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3866000000000001e-05 2024-09-16T03:40:56.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.440 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.440 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.498525+0000", 2024-09-16T03:40:56.440 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048063230000000004 2024-09-16T03:40:56.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.440 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.498570+0000", 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5058999999999997e-05 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=31, reqid=osd.8.0:31, at_version=49'22, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 27.936908271, 2024-09-16T03:40:56.441 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059536579999999997, 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.442 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:40:56.443 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.443 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.443 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.443 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495459+0000", 2024-09-16T03:40:56.443 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3799999999999996e-07 2024-09-16T03:40:56.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.443 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495484+0000", 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5601000000000001e-05 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495505+0000", 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0460999999999999e-05 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495514+0000", 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1390000000000001e-06 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495592+0000", 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7877e-05 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.445 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495607+0000", 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5576000000000001e-05 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501362+0000", 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057546280000000003 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501411+0000", 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9438000000000003e-05 2024-09-16T03:40:56.446 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 23.927618834, 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068498659999999996, 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.447 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.448 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.448 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.448 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.448 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:40:56.448 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.448 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.448 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.448 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.448 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:40:56.448 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504748+0000", 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4499999999999997e-07 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504768+0000", 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0582999999999999e-05 2024-09-16T03:40:56.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.450 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.450 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.450 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504782+0000", 2024-09-16T03:40:56.450 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3883000000000001e-05 2024-09-16T03:40:56.450 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.451 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.451 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.451 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504790+0000", 2024-09-16T03:40:56.451 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3389999999999993e-06 2024-09-16T03:40:56.451 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.451 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.451 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.451 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504921+0000", 2024-09-16T03:40:56.451 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013093800000000001 2024-09-16T03:40:56.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.452 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.452 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504930+0000", 2024-09-16T03:40:56.452 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7499999999999992e-06 2024-09-16T03:40:56.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.452 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511566+0000", 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066358449999999996 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511597+0000", 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0883000000000002e-05 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=29, reqid=client.4615.0:356, at_version=53'10, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.506019+0000", 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 23.926346654, 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0056103890000000003, 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.454 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506019+0000", 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506019+0000", 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506020+0000", 2024-09-16T03:40:56.455 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506032+0000", 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2446000000000001e-05 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506042+0000", 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6579999999999997e-06 2024-09-16T03:40:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506047+0000", 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5199999999999997e-06 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506099+0000", 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1518999999999999e-05 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.457 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506110+0000", 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0496000000000001e-05 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511600+0000", 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054906190000000004 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.458 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511630+0000", 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9447000000000001e-05 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=30, reqid=client.4615.0:357, at_version=53'12, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.508274+0000", 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 23.924091702999998, 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0056031689999999999, 2024-09-16T03:40:56.459 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.460 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.460 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.460 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.460 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.460 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.460 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508274+0000", 2024-09-16T03:40:56.460 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.460 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.460 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508274+0000", 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508275+0000", 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2500000000000005e-07 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.461 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508534+0000", 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025936500000000001 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508549+0000", 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4664000000000001e-05 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508557+0000", 2024-09-16T03:40:56.462 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4079999999999998e-06 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508619+0000", 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1467e-05 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.508628+0000", 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2210000000000003e-06 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.464 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.464 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513823+0000", 2024-09-16T03:40:56.464 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051947820000000002 2024-09-16T03:40:56.464 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.464 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.464 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.464 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513877+0000", 2024-09-16T03:40:56.464 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4537000000000003e-05 2024-09-16T03:40:56.464 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.464 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 53/31 ECSubWrite(tid=20, reqid=client.4615.0:376, at_version=53'13, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.544496+0000", 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 23.887869390999999, 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059495989999999999, 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.465 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544496+0000", 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544496+0000", 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.466 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544498+0000", 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3519999999999999e-06 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545106+0000", 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00060862000000000004 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545125+0000", 2024-09-16T03:40:56.467 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9052999999999999e-05 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545136+0000", 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0125e-05 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545201+0000", 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5769e-05 2024-09-16T03:40:56.468 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.469 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.469 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.469 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545212+0000", 2024-09-16T03:40:56.469 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0611000000000001e-05 2024-09-16T03:40:56.469 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.469 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.469 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.469 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550406+0000", 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051939519999999999 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550446+0000", 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0117000000000002e-05 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.470 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 53/31 ECSubWrite(tid=21, reqid=client.4615.0:377, at_version=53'14, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.545414+0000", 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 23.886951801999999, 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050486039999999999, 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.471 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545414+0000", 2024-09-16T03:40:56.472 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.472 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.472 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.472 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.472 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545414+0000", 2024-09-16T03:40:56.472 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.472 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.472 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.472 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.472 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545415+0000", 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1699999999999997e-07 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545439+0000", 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4539999999999999e-05 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545452+0000", 2024-09-16T03:40:56.473 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2150000000000001e-05 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545460+0000", 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1929999999999993e-06 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545704+0000", 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000243985 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.474 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545714+0000", 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0103e-05 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550450+0000", 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047365330000000002 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.475 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550463+0000", 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2283000000000001e-05 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 53/31 ECSubWrite(tid=93, reqid=client.4615.0:389, at_version=53'28, trim_to=0'0, roll_forward_to=42'25))", 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.554348+0000", 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 23.878018146999999, 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047364149999999999, 2024-09-16T03:40:56.476 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554348+0000", 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554348+0000", 2024-09-16T03:40:56.477 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554349+0000", 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.16e-07 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554719+0000", 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000370892 2024-09-16T03:40:56.478 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554732+0000", 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2918999999999999e-05 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554741+0000", 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4230000000000005e-06 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.479 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554821+0000", 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9753999999999996e-05 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.554830+0000", 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1630000000000002e-06 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.480 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.559049+0000", 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042190839999999997 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.559084+0000", 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5264e-05 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.481 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 63/31 ECSubWrite(tid=140, reqid=osd.0.0:140, at_version=63'56, trim_to=0'0, roll_forward_to=60'52))", 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.476911+0000", 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.955454731, 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0063366669999999998, 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476911+0000", 2024-09-16T03:40:56.482 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476911+0000", 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476912+0000", 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4099999999999998e-07 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.483 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476932+0000", 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0027e-05 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476942+0000", 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7429999999999997e-06 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.484 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476950+0000", 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3510000000000002e-06 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.478026+0000", 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001076411 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.478036+0000", 2024-09-16T03:40:56.485 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5750000000000007e-06 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483217+0000", 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051814000000000001 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483248+0000", 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0419000000000001e-05 2024-09-16T03:40:56.486 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 63/31 ECSubWrite(tid=31, reqid=osd.15.0:31, at_version=63'24, trim_to=0'0, roll_forward_to=60'22))", 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.955431059, 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0062644750000000003, 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.487 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6899999999999999e-07 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476965+0000", 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9714999999999999e-05 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476988+0000", 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3297000000000001e-05 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476998+0000", 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0254999999999999e-05 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.477097+0000", 2024-09-16T03:40:56.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8939999999999998e-05 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.477108+0000", 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.039e-05 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483157+0000", 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060496719999999999 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.491 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483199+0000", 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1937000000000002e-05 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 63/31 ECSubWrite(tid=59, reqid=osd.12.0:59, at_version=63'27, trim_to=0'0, roll_forward_to=60'25))", 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.480620+0000", 2024-09-16T03:40:56.492 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.951745709000001, 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054344809999999997, 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480620+0000", 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480620+0000", 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480620+0000", 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3999999999999998e-07 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480648+0000", 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.739e-05 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480659+0000", 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1484999999999999e-05 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480668+0000", 2024-09-16T03:40:56.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2180000000000005e-06 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480738+0000", 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9628999999999994e-05 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480751+0000", 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3139000000000001e-05 2024-09-16T03:40:56.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486003+0000", 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052515549999999998 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486055+0000", 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1824999999999999e-05 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.497 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 63/31 ECSubWrite(tid=52, reqid=osd.8.0:51, at_version=0'0, trim_to=0'0, roll_forward_to=63'40))", 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.481294+0000", 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.95107196, 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047886339999999999, 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.481294+0000", 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.481294+0000", 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.481298+0000", 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5310000000000003e-06 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.481308+0000", 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6760000000000002e-06 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.481323+0000", 2024-09-16T03:40:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5000999999999999e-05 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.481332+0000", 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6910000000000003e-06 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.481402+0000", 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0722999999999994e-05 2024-09-16T03:40:56.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.481412+0000", 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7329999999999998e-06 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486059+0000", 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046463420000000004 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486083+0000", 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3937000000000001e-05 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 63/31 ECSubWrite(tid=60, reqid=osd.12.0:59, at_version=0'0, trim_to=0'0, roll_forward_to=63'27))", 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.486758+0000", 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.945607901000001, 2024-09-16T03:40:56.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044840360000000003, 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486758+0000", 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.504 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486758+0000", 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486759+0000", 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8400000000000003e-07 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486771+0000", 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2337999999999999e-05 2024-09-16T03:40:56.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486783+0000", 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2217e-05 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486793+0000", 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3139999999999993e-06 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486861+0000", 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8393999999999996e-05 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.486873+0000", 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1612e-05 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.491208+0000", 2024-09-16T03:40:56.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004335429 2024-09-16T03:40:56.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.491242+0000", 2024-09-16T03:40:56.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3948000000000001e-05 2024-09-16T03:40:56.508 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.508 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 73/31 ECSubWrite(tid=120, reqid=osd.12.0:119, at_version=0'0, trim_to=0'0, roll_forward_to=73'56))", 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.491024+0000", 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.9413415380000001, 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005258548, 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491024+0000", 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491024+0000", 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.510 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491025+0000", 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6100000000000001e-07 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491034+0000", 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8060000000000001e-06 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491042+0000", 2024-09-16T03:40:56.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7139999999999999e-06 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491049+0000", 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5860000000000001e-06 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491146+0000", 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7480999999999997e-05 2024-09-16T03:40:56.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491157+0000", 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0910000000000001e-05 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496241+0000", 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050839509999999997 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496283+0000", 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1539000000000003e-05 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 73/31 ECSubWrite(tid=41, reqid=osd.15.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=73'31))", 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.497839+0000", 2024-09-16T03:40:56.514 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.9345264179999999, 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060867669999999999, 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497839+0000", 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497839+0000", 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497840+0000", 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0100000000000001e-06 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.516 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497854+0000", 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3358e-05 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497871+0000", 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6994999999999999e-05 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497880+0000", 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3589999999999997e-06 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497967+0000", 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7106000000000003e-05 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497978+0000", 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0370999999999999e-05 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.503877+0000", 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058990930000000002 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.503926+0000", 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9475000000000003e-05 2024-09-16T03:40:56.519 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 74/55 ECSubWrite(tid=30, reqid=osd.3.0:30, at_version=74'47, trim_to=0'0, roll_forward_to=73'39))", 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.535733+0000", 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8966331379999999, 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066050730000000004, 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.520 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.521 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.521 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535733+0000", 2024-09-16T03:40:56.521 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.521 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.521 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535733+0000", 2024-09-16T03:40:56.521 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535734+0000", 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.246e-06 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535760+0000", 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5706000000000001e-05 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.522 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535773+0000", 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3682e-05 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535782+0000", 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9090000000000006e-06 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.536271+0000", 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00048888700000000002 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.536282+0000", 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.078e-05 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542291+0000", 2024-09-16T03:40:56.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060094379999999998 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542338+0000", 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6424999999999997e-05 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.525 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 74/31 ECSubWrite(tid=129, reqid=client.4615.0:681, at_version=74'64, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.537734+0000", 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8946318409999998, 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051788880000000004, 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.537734+0000", 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.537734+0000", 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.537735+0000", 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5300000000000003e-07 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.537754+0000", 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9389e-05 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.537768+0000", 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.349e-05 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.537777+0000", 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3789999999999995e-06 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.537845+0000", 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7890000000000002e-05 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.537854+0000", 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8130000000000002e-06 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542866+0000", 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050123900000000002 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542913+0000", 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6684000000000002e-05 2024-09-16T03:40:56.530 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 74/31 ECSubWrite(tid=130, reqid=osd.12.0:130, at_version=74'66, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.539468+0000", 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8928978380000001, 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054691619999999996, 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.531 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539468+0000", 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539468+0000", 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539469+0000", 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539498+0000", 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9277000000000001e-05 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539515+0000", 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6999000000000001e-05 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539524+0000", 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0100000000000001e-06 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.534 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539597+0000", 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3059000000000006e-05 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539607+0000", 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8430000000000005e-06 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544897+0000", 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052899949999999996 2024-09-16T03:40:56.535 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544937+0000", 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0083000000000002e-05 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 74/31 ECSubWrite(tid=131, reqid=osd.12.0:131, at_version=74'68, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T03:40:56.536 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.539984+0000", 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.892381855, 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078357930000000006, 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539984+0000", 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539984+0000", 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539985+0000", 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0299999999999997e-07 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540008+0000", 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2765999999999999e-05 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540022+0000", 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4455e-05 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.539 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540031+0000", 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2e-06 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540256+0000", 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000224391 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540280+0000", 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3952e-05 2024-09-16T03:40:56.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.541 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.541 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.541 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547770+0000", 2024-09-16T03:40:56.541 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007489941 2024-09-16T03:40:56.541 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.541 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.541 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.541 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547820+0000", 2024-09-16T03:40:56.541 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0185000000000001e-05 2024-09-16T03:40:56.541 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 74/31 ECSubWrite(tid=195, reqid=osd.0.0:194, at_version=0'0, trim_to=0'0, roll_forward_to=74'100))", 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.542567+0000", 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8897987839999999, 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052834300000000004, 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.542 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542567+0000", 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542567+0000", 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.543 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542568+0000", 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1999999999999999e-06 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542581+0000", 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2322e-05 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542595+0000", 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4052e-05 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542604+0000", 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6690000000000001e-06 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542673+0000", 2024-09-16T03:40:56.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8486999999999997e-05 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542682+0000", 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1670000000000005e-06 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547799+0000", 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051172559999999997 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.546 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.547 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.547 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547850+0000", 2024-09-16T03:40:56.547 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1276999999999997e-05 2024-09-16T03:40:56.547 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.547 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.547 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.547 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.547 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.547 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:56.549 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:56.552 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:56.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:40:56.554 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T03:40:56.557 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:56.558 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:56.559 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:56.560 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:40:56.546+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:40:56.560 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:56.562 INFO:tasks.rados.rados.0.smithi121.stdout:271: done (2 left) 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:272: done (1 left) 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:273: done (0 left) 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:274: copy_from oid 22 from oid 32 current snap is 26 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:275: read oid 49 snap -1 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:275: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 13 current snap is 26 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:276: seq_num 127 ranges {1900544=589824,2490368=753664,3244032=475136,3719168=327680} 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:276: writing smithi12138290-13 from 1900544 to 2490368 tid 1 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:276: writing smithi12138290-13 from 2490368 to 3244032 tid 2 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:276: writing smithi12138290-13 from 3244032 to 3719168 tid 3 2024-09-16T03:40:56.563 INFO:tasks.rados.rados.0.smithi121.stdout:276: writing smithi12138290-13 from 3719168 to 4046848 tid 4 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:277: delete oid 44 current snap is 26 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:275: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:274: finishing copy_from racing read to smithi12138290-22 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:274: finishing copy_from to smithi12138290-22 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 22 v 76 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:276: finishing write tid 1 to smithi12138290-13 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:276: finishing write tid 2 to smithi12138290-13 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:276: finishing write tid 3 to smithi12138290-13 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:274: done (3 left) 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:275: done (2 left) 2024-09-16T03:40:56.564 INFO:tasks.rados.rados.0.smithi121.stdout:277: done (1 left) 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:278: rollback oid 11 current snap is 26 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 11 to 16 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:276: finishing write tid 4 to smithi12138290-13 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 5 current snap is 26 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:279: seq_num 129 ranges {1589248=622592,2211840=655360,2867200=688128,3555328=212992} 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:279: writing smithi12138290-5 from 1589248 to 2211840 tid 1 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:279: writing smithi12138290-5 from 2211840 to 2867200 tid 2 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:279: writing smithi12138290-5 from 2867200 to 3555328 tid 3 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:279: writing smithi12138290-5 from 3555328 to 3768320 tid 4 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 12 current snap is 26 2024-09-16T03:40:56.565 INFO:tasks.rados.rados.0.smithi121.stdout:276: finishing write tid 5 to smithi12138290-13 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:280: seq_num 130 ranges {1376256=638976,2015232=671744,2686976=802816,3489792=81920} 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:280: writing smithi12138290-12 from 1376256 to 2015232 tid 1 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:280: writing smithi12138290-12 from 2015232 to 2686976 tid 2 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:280: writing smithi12138290-12 from 2686976 to 3489792 tid 3 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:280: writing smithi12138290-12 from 3489792 to 3571712 tid 4 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:281: delete oid 35 current snap is 26 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:276: finishing write tid 6 to smithi12138290-13 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:276: oid 13 updating version 0 to 74 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:276: oid 13 updating version 74 to 76 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:276: oid 13 version 76 is already newer than 75 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:276: oid 13 updating version 76 to 78 2024-09-16T03:40:56.566 INFO:tasks.rados.rados.0.smithi121.stdout:276: oid 13 version 78 is already newer than 77 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 13 v 78 (ObjNum 127 snap 26 seq_num 127) dirty exists 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:276: left oid 13 (ObjNum 127 snap 26 seq_num 127) 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:278: finishing rollback tid 1 to smithi12138290-11 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:278: finishing rollback tid 2 to smithi12138290-11 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 11 v 101 (ObjNum 77 snap 11 seq_num 77) dirty exists 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:279: finishing write tid 1 to smithi12138290-5 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:279: finishing write tid 2 to smithi12138290-5 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:279: finishing write tid 3 to smithi12138290-5 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:279: finishing write tid 4 to smithi12138290-5 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:279: finishing write tid 5 to smithi12138290-5 2024-09-16T03:40:56.567 INFO:tasks.rados.rados.0.smithi121.stdout:279: finishing write tid 6 to smithi12138290-5 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:279: oid 5 updating version 0 to 34 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:279: oid 5 updating version 34 to 38 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:279: oid 5 version 38 is already newer than 37 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:279: oid 5 version 38 is already newer than 36 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:279: oid 5 version 38 is already newer than 35 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 5 v 38 (ObjNum 129 snap 26 seq_num 129) dirty exists 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:279: left oid 5 (ObjNum 129 snap 26 seq_num 129) 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:280: finishing write tid 1 to smithi12138290-12 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:280: finishing write tid 2 to smithi12138290-12 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:280: finishing write tid 3 to smithi12138290-12 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:280: finishing write tid 4 to smithi12138290-12 2024-09-16T03:40:56.568 INFO:tasks.rados.rados.0.smithi121.stdout:276: done (4 left) 2024-09-16T03:40:56.569 INFO:tasks.rados.rados.0.smithi121.stdout:278: done (3 left) 2024-09-16T03:40:56.569 INFO:tasks.rados.rados.0.smithi121.stdout:279: done (2 left) 2024-09-16T03:40:56.569 INFO:tasks.rados.rados.0.smithi121.stdout:281: done (1 left) 2024-09-16T03:40:56.569 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 30 current snap is 26 2024-09-16T03:40:56.569 INFO:tasks.rados.rados.0.smithi121.stdout:282: seq_num 132 ranges {3129344=671744,3801088=573440,4374528=753664,5128192=229376} 2024-09-16T03:40:56.572 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:56.574 INFO:tasks.rados.rados.0.smithi121.stdout:282: writing smithi12138290-30 from 3129344 to 3801088 tid 1 2024-09-16T03:40:56.578 INFO:tasks.rados.rados.0.smithi121.stdout:282: writing smithi12138290-30 from 3801088 to 4374528 tid 2 2024-09-16T03:40:56.582 INFO:tasks.rados.rados.0.smithi121.stdout:282: writing smithi12138290-30 from 4374528 to 5128192 tid 3 2024-09-16T03:40:56.585 INFO:tasks.rados.rados.0.smithi121.stdout:282: writing smithi12138290-30 from 5128192 to 5357568 tid 4 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:283: rmattr oid 15 current snap is 26 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:283: done (2 left) 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:284: read oid 39 snap -1 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:284: expect (ObjNum 101 snap 20 seq_num 101) 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:285: rollback oid 5 current snap is 26 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 5 to 23 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:280: finishing write tid 5 to smithi12138290-12 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:280: finishing write tid 6 to smithi12138290-12 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:280: oid 12 updating version 0 to 79 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:280: oid 12 updating version 79 to 81 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:280: oid 12 version 81 is already newer than 80 2024-09-16T03:40:56.586 INFO:tasks.rados.rados.0.smithi121.stdout:280: oid 12 version 81 is already newer than 78 2024-09-16T03:40:56.587 INFO:tasks.rados.rados.0.smithi121.stdout:280: oid 12 version 81 is already newer than 77 2024-09-16T03:40:56.587 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 81 (ObjNum 130 snap 26 seq_num 130) dirty exists 2024-09-16T03:40:56.587 INFO:tasks.rados.rados.0.smithi121.stdout:280: left oid 12 (ObjNum 130 snap 26 seq_num 130) 2024-09-16T03:40:56.587 INFO:tasks.rados.rados.0.smithi121.stdout:282: finishing write tid 1 to smithi12138290-30 2024-09-16T03:40:56.587 INFO:tasks.rados.rados.0.smithi121.stdout:280: done (3 left) 2024-09-16T03:40:56.587 INFO:tasks.rados.rados.0.smithi121.stdout:286: read oid 4 snap 25 2024-09-16T03:40:56.587 INFO:tasks.rados.rados.0.smithi121.stdout:286: expect (ObjNum 79 snap 11 seq_num 79) 2024-09-16T03:40:56.587 INFO:tasks.rados.rados.0.smithi121.stdout:287: read oid 22 snap -1 2024-09-16T03:40:56.587 INFO:tasks.rados.rados.0.smithi121.stdout:287: expect (ObjNum 114 snap 23 seq_num 114) 2024-09-16T03:40:56.587 INFO:tasks.rados.rados.0.smithi121.stdout:288: snap_remove snap 20 2024-09-16T03:40:56.594 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:56.594 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:56.610 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:56.610 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:56.627 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:56.627 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:56.645 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:56.645 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:56.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:56.643+0000 7f12a6d9d640 0 -- 172.21.15.76:0/1250625273 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f12a82211f0 msgr2=0x7f12a82235e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:56.649 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:56.646+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:56.662 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:56.662 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:56.662 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:56.662 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:56.663 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:56.663 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:56.671 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:40:56.750 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:56.748+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:56.774 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:56.775 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:56.775 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:56.775 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:56.775 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:56.783 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:40:56.851 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:56.849+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:56.929 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:56.929 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:56.929 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:270 3.1s0 3:82a78c8d:::smithi12138290-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 35.298945762999999, 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086784840000000002, 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:56.930 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 270 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627878+0000", 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8299999999999998e-07 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627880+0000", 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.716e-06 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627895+0000", 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4904e-05 2024-09-16T03:40:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.627900+0000", 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4990000000000002e-06 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630550+0000", 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0026501960000000001 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.630645+0000", 2024-09-16T03:40:56.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4790999999999994e-05 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635427+0000", 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047821740000000001 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.635478+0000", 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0603000000000003e-05 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:21.636556+0000", 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001078318 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:280 3.1s0 3:8df00818:::smithi12138290-14:head [copy-from ver 2] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.569179+0000", 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 34.357644651999998, 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080900399999999997, 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.935 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 280 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569179+0000", 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.936 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569179+0000", 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569181+0000", 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3970000000000001e-06 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569193+0000", 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1744e-05 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569210+0000", 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7286e-05 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569223+0000", 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2852e-05 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569304+0000", 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0815000000000005e-05 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.569310+0000", 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9800000000000003e-06 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571309+0000", 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001999158 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.571423+0000", 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011391399999999999 2024-09-16T03:40:56.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575663+0000", 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042401619999999996 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.576691+0000", 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010281489999999999 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:56.940 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577144+0000", 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00045261500000000002 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577269+0000", 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012496799999999999 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:301 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~491520] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:40:56.941 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.443965418000001, 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013883662999999999, 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 301 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:40:56.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.943 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.943 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:40:56.943 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.943 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.943 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482860+0000", 2024-09-16T03:40:56.943 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.911e-06 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483803+0000", 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00094328700000000001 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483820+0000", 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6249e-05 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483828+0000", 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.704e-06 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483895+0000", 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6839000000000002e-05 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483902+0000", 2024-09-16T03:40:56.945 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.485797+0000", 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018946049999999999 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.485916+0000", 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011867 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.946 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487208+0000", 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012921110000000001 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493186+0000", 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059786010000000001 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496639+0000", 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034530749999999999 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.947 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.948 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496742+0000", 2024-09-16T03:40:56.948 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102394 2024-09-16T03:40:56.948 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.948 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.948 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.948 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.948 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:302 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~770048] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:56.948 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.439686145, 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014729638999999999, 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 302 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.949 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487139+0000", 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.378e-06 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488116+0000", 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00097674700000000003 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488127+0000", 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1817999999999999e-05 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488135+0000", 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6620000000000005e-06 2024-09-16T03:40:56.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489443+0000", 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001308303 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489564+0000", 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000120664 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490525+0000", 2024-09-16T03:40:56.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00096115100000000004 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493256+0000", 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027313429999999998 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501784+0000", 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0085277840000000001 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501867+0000", 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2788999999999996e-05 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:303 3.1s0 3:8a735ecd:::smithi12138290-30:head [setxattr _header (57) in=7b] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.438674609, 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013931318, 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.954 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 303 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:40:56.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488150+0000", 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000001e-07 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488152+0000", 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.509e-06 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.956 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488157+0000", 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5739999999999997e-06 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488160+0000", 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.349e-06 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491536+0000", 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033756799999999998 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491638+0000", 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010186799999999999 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492139+0000", 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050129999999999999 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493257+0000", 2024-09-16T03:40:56.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001118551 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501989+0000", 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087319819999999992 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502080+0000", 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0827000000000003e-05 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.959 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:304 3.1s0 3:8a735ecd:::smithi12138290-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.438657631000002, 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.024338029000000001, 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 304 2024-09-16T03:40:56.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.53e-07 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488169+0000", 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3309999999999998e-06 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488173+0000", 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488176+0000", 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0539999999999999e-06 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492435+0000", 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042594399999999998 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492534+0000", 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8558000000000001e-05 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505269+0000", 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012735076 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505321+0000", 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1977000000000001e-05 2024-09-16T03:40:56.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.512504+0000", 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0071833060000000004 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:309 3.1s0 3:82a78c8d:::smithi12138290-1:head [copy-from ver 4] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.435998177999998, 2024-09-16T03:40:56.967 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020962743999999998, 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 309 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:40:56.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490827+0000", 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490832+0000", 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4550000000000003e-06 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490845+0000", 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2951e-05 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490873+0000", 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7566999999999999e-05 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.970 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492571+0000", 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001698359 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492650+0000", 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8858999999999998e-05 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503971+0000", 2024-09-16T03:40:56.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011320805 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511333+0000", 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073624160000000001 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511735+0000", 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00040226999999999998 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.972 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511788+0000", 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2757999999999999e-05 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:310 3.1s0 3:82a78c8d:::smithi12138290-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.435932199, 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014337858, 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 310 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:40:56.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490892+0000", 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.37e-07 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490896+0000", 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3710000000000002e-06 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490932+0000", 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5796000000000001e-05 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490941+0000", 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8589999999999994e-06 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493109+0000", 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021679310000000001 2024-09-16T03:40:56.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for blocked object", 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493167+0000", 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7572999999999999e-05 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505077+0000", 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011909968999999999 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505164+0000", 2024-09-16T03:40:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7409000000000003e-05 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505229+0000", 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5213000000000001e-05 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=22, reqid=client.4615.0:308, at_version=49'14, trim_to=0'0, roll_forward_to=42'12))", 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.494598+0000", 2024-09-16T03:40:56.978 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.432225449000001, 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084633480000000007, 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494598+0000", 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494598+0000", 2024-09-16T03:40:56.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494599+0000", 2024-09-16T03:40:56.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6899999999999998e-07 2024-09-16T03:40:56.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494838+0000", 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000239409 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494848+0000", 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8859999999999999e-06 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494855+0000", 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9739999999999996e-06 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494918+0000", 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3478000000000004e-05 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.494925+0000", 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1940000000000003e-06 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503029+0000", 2024-09-16T03:40:56.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081040469999999996 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503061+0000", 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2891000000000001e-05 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 49/31 ECSubWriteReply(tid=34, last_complete=49'24, committed=1, applied=1))", 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.503107+0000", 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.423716793000001, 2024-09-16T03:40:56.983 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081770950000000005, 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503107+0000", 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503107+0000", 2024-09-16T03:40:56.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503108+0000", 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.61e-07 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503112+0000", 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9239999999999998e-06 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503122+0000", 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4599999999999992e-06 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503129+0000", 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6540000000000002e-06 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511256+0000", 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081276979999999992 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511284+0000", 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7597999999999999e-05 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 49/31 ECSubWriteReply(tid=34, last_complete=49'24, committed=1, applied=1))", 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.503421+0000", 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.423402792000001, 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078850580000000003, 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.987 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503421+0000", 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503421+0000", 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503422+0000", 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503426+0000", 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4689999999999999e-06 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503439+0000", 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2802e-05 2024-09-16T03:40:56.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503447+0000", 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0669999999999992e-06 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511292+0000", 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078452020000000008 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511306+0000", 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3689000000000001e-05 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 49/31 ECSubWriteReply(tid=34, last_complete=49'24, committed=1, applied=1))", 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.423349807000001, 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00786215, 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:56.991 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503474+0000", 2024-09-16T03:40:56.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3099999999999998e-07 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503477+0000", 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1520000000000001e-06 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503488+0000", 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0387000000000001e-05 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503494+0000", 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9850000000000003e-06 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511323+0000", 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078294260000000004 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511336+0000", 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2669e-05 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:56.994 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:337 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~606208] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.492469+0000", 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.434354585000001, 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0083634099999999999, 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:56.995 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 337 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.492469+0000", 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.492469+0000", 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.492470+0000", 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.511e-06 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493117+0000", 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00064688400000000002 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493129+0000", 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.203e-05 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493135+0000", 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8139999999999997e-06 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493197+0000", 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2141e-05 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.493305+0000", 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107426 2024-09-16T03:40:56.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.494433+0000", 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001128471 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496206+0000", 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001772608 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:56.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500752+0000", 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045464759999999998 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500832+0000", 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0049000000000006e-05 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:338 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~720896] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.496989+0000", 2024-09-16T03:40:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.429834068000002, 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0077073139999999998, 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 338 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.001 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496989+0000", 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496989+0000", 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.496991+0000", 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0449999999999999e-06 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497710+0000", 2024-09-16T03:40:57.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00071904 2024-09-16T03:40:57.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497719+0000", 2024-09-16T03:40:57.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3409999999999992e-06 2024-09-16T03:40:57.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497723+0000", 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4610000000000001e-06 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497761+0000", 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7469000000000003e-05 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497832+0000", 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1612999999999994e-05 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.498416+0000", 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00058368299999999999 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.501013+0000", 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025971829999999999 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.504614+0000", 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036004779999999998 2024-09-16T03:40:57.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.504697+0000", 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3001e-05 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:341 3.1s0 3:8a735ecd:::smithi12138290-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.429046927999998, 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.008644522, 2024-09-16T03:40:57.006 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 341 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497777+0000", 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4400000000000001e-07 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497779+0000", 2024-09-16T03:40:57.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7260000000000001e-06 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497783+0000", 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9509999999999999e-06 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.497786+0000", 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8629999999999999e-06 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.500932+0000", 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031462389999999999 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.501005+0000", 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3546000000000004e-05 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.505207+0000", 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042014330000000001 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.505269+0000", 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2337999999999998e-05 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:31.506421+0000", 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001152082 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.012 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds6 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:40:57.012 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.504801+0000", 2024-09-16T03:40:57.012 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.422022276, 2024-09-16T03:40:57.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0082787090000000004, 2024-09-16T03:40:57.012 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.012 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.012 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.012 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.012 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504801+0000", 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504801+0000", 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504802+0000", 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1600000000000001e-07 2024-09-16T03:40:57.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504821+0000", 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9117999999999999e-05 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504833+0000", 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.151e-05 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504840+0000", 2024-09-16T03:40:57.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5959999999999998e-06 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504899+0000", 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8693000000000002e-05 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504909+0000", 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6590000000000002e-06 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513048+0000", 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081393769999999997 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.513080+0000", 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2039999999999998e-05 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:593 3.1s0 3:82a78c8d:::smithi12138290-1:head [stat,delete] snapc 18=[18,17,16,15,13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T03:40:57.016 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:48.498370+0000", 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 8.4284536810000006, 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012012600999999999, 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 593 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.017 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498370+0000", 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498370+0000", 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498372+0000", 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9589999999999998e-06 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498379+0000", 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9619999999999996e-06 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498396+0000", 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6801000000000001e-05 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498405+0000", 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5380000000000008e-06 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498490+0000", 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4481999999999995e-05 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498497+0000", 2024-09-16T03:40:57.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2989999999999999e-06 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.506967+0000", 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084698340000000007 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.507051+0000", 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3943999999999995e-05 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.507813+0000", 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076275500000000005 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.508730+0000", 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00091662999999999998 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.510309+0000", 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001578772 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.510382+0000", 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3625000000000001e-05 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/31 ECSubWrite(tid=131, reqid=osd.12.0:131, at_version=74'68, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.539975+0000", 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.3868485650000002, 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0077020470000000001, 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.023 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539975+0000", 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539975+0000", 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539976+0000", 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5000000000000001e-07 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539998+0000", 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2045999999999999e-05 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540014+0000", 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5603999999999998e-05 2024-09-16T03:40:57.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540023+0000", 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9670000000000007e-06 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540525+0000", 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050209400000000002 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540534+0000", 2024-09-16T03:40:57.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5030000000000003e-06 2024-09-16T03:40:57.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547634+0000", 2024-09-16T03:40:57.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0070993719999999996 2024-09-16T03:40:57.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547677+0000", 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3510999999999997e-05 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:765 3.1s0 3:83f0987c:::smithi12138290-35:head [delete] snapc 1b=[1b,1a,19,16,12,11,10,b,8] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:56.544145+0000", 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.38267813899999997, 2024-09-16T03:40:57.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0097551189999999996, 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 765 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.544145+0000", 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.544145+0000", 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.544148+0000", 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9799999999999998e-06 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.544163+0000", 2024-09-16T03:40:57.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5003e-05 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.544197+0000", 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3299000000000001e-05 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.544219+0000", 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2212e-05 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.544350+0000", 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013076299999999999 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.544448+0000", 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8027000000000002e-05 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.544958+0000", 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00051057299999999995 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.545566+0000", 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00060739500000000005 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.553812+0000", 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0082460369999999995 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.553901+0000", 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8830000000000002e-05 2024-09-16T03:40:57.033 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:768 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~753664] snapc 1b=[1b,1a,19,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:56.582048+0000", 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.34477574300000002, 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078174879999999992, 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.034 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 768 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.582048+0000", 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.582048+0000", 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.582049+0000", 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.22e-06 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.582743+0000", 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00069351099999999995 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.582751+0000", 2024-09-16T03:40:57.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.582757+0000", 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3539999999999999e-06 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.582880+0000", 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012371600000000001 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.582970+0000", 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9738e-05 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.583794+0000", 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00082418699999999999 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.584630+0000", 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00083556400000000001 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.589816+0000", 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051865890000000001 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:56.589865+0000", 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8986999999999997e-05 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.039 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:57.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:40:57.041 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:57.039+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:57.141 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:57.140+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:57.143 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:57.144 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:57.144 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:57.144 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:57.152 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:40:57.242 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:57.240+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:57.255 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:57.255 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:57.255 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:57.255 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:57.255 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:57.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:40:57.349 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:57.347+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:57.410 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:57.410 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:57.410 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:57.410 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:57.410 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.410 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:282 3.4s0 3:2307c0f5:::smithi12138290-28:head [append 0~458752] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:40:57.410 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 34.832573369999999, 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0093867299999999994, 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 282 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.411 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574793+0000", 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5799999999999999e-06 2024-09-16T03:40:57.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575536+0000", 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00074309099999999998 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575567+0000", 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0382000000000002e-05 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575590+0000", 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3215000000000001e-05 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575665+0000", 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5462000000000004e-05 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575671+0000", 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6969999999999998e-06 2024-09-16T03:40:57.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577693+0000", 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020222439999999999 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577796+0000", 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010306000000000001 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578998+0000", 2024-09-16T03:40:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012013880000000001 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.583553+0000", 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045549340000000001 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584072+0000", 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00051922199999999996 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584177+0000", 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105455 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:311 3.bs0 3:db21b75b:::smithi12138290-3:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.916421268000001, 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013432349, 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.417 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 311 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.418 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490945+0000", 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.419e-06 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.419 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490953+0000", 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7379999999999994e-06 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490973+0000", 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0460999999999999e-05 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490984+0000", 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0857e-05 2024-09-16T03:40:57.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491053+0000", 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8351999999999999e-05 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491061+0000", 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3569999999999998e-06 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493588+0000", 2024-09-16T03:40:57.421 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025267430000000001 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493683+0000", 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4876000000000004e-05 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496393+0000", 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027106500000000002 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504203+0000", 2024-09-16T03:40:57.422 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078100080000000002 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504276+0000", 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2601000000000005e-05 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504375+0000", 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9287e-05 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:312 3.bs0 3:db21b75b:::smithi12138290-3:head [rollback a] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.916355642999999, 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013925697000000001, 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 312 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.424 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.425 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491009+0000", 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0200000000000001e-07 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491013+0000", 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8160000000000004e-06 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491022+0000", 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6649999999999992e-06 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.426 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491028+0000", 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4760000000000003e-06 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491124+0000", 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6192000000000003e-05 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491132+0000", 2024-09-16T03:40:57.427 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6539999999999999e-06 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499113+0000", 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079811880000000002 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499184+0000", 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0749999999999999e-05 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.428 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499557+0000", 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037325200000000002 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504380+0000", 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048224590000000003 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504852+0000", 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047194899999999999 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504934+0000", 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2594000000000002e-05 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:313 3.bs0 3:db21b75b:::smithi12138290-3:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.916314002, 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016369406999999999, 2024-09-16T03:40:57.430 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 313 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.431 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491051+0000", 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.82e-07 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.432 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491054+0000", 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5260000000000002e-06 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491063+0000", 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.354e-06 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491069+0000", 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2299999999999996e-06 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.433 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491177+0000", 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000108285 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491184+0000", 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0489999999999998e-06 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501958+0000", 2024-09-16T03:40:57.434 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010773448 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502064+0000", 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106661 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502545+0000", 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000481077 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.435 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504381+0000", 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018352970000000001 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.507349+0000", 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002968666 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.507420+0000", 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0232000000000001e-05 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.436 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:369 3.4s0 3:2ef36fe6:::smithi12138290-31:head [append 0~131072] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.877783516000001, 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090412610000000001, 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.437 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 369 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.438 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2399999999999998e-07 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529661+0000", 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9846999999999996e-05 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529666+0000", 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1919999999999996e-06 2024-09-16T03:40:57.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529670+0000", 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3830000000000001e-06 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.530925+0000", 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012551730000000001 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.440 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.531047+0000", 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121934 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.531824+0000", 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000777384 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.534187+0000", 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023626609999999998 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.441 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538565+0000", 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043779170000000003 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538622+0000", 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7246e-05 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.442 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:370 3.4s0 3:2ef36fe6:::smithi12138290-31:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.529676+0000", 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.877687712, 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092009889999999997, 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 370 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.443 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529676+0000", 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529676+0000", 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529677+0000", 2024-09-16T03:40:57.444 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9099999999999999e-07 2024-09-16T03:40:57.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.445 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.445 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.445 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529679+0000", 2024-09-16T03:40:57.445 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7799999999999999e-06 2024-09-16T03:40:57.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.445 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.445 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529682+0000", 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4209999999999999e-06 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529685+0000", 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3459999999999998e-06 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533114+0000", 2024-09-16T03:40:57.446 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034284559999999999 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533215+0000", 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010074200000000001 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533660+0000", 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044525100000000002 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.447 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.534705+0000", 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001044971 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538781+0000", 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040766639999999998 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538877+0000", 2024-09-16T03:40:57.448 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5966999999999998e-05 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:371 3.4s0 3:2ef36fe6:::smithi12138290-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.877674931000001, 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0095505, 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.449 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 371 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.450 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7799999999999997e-07 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529691+0000", 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2330000000000001e-06 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.451 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529694+0000", 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5489999999999998e-06 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529697+0000", 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.706e-06 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533902+0000", 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004204564 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533982+0000", 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0399000000000001e-05 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.539228+0000", 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052458579999999999 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.539240+0000", 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1912999999999999e-05 2024-09-16T03:40:57.453 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:381 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.542096+0000", 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.865267791000001, 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086742629999999998, 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.454 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 381 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542096+0000", 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.455 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542096+0000", 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542097+0000", 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2999999999999999e-07 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542098+0000", 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.22e-06 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542101+0000", 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2590000000000001e-06 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.542104+0000", 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7939999999999998e-06 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544054+0000", 2024-09-16T03:40:57.457 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001950011 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544139+0000", 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4691000000000006e-05 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544560+0000", 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00042157700000000002 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.458 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.547938+0000", 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033775860000000001 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550714+0000", 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027767080000000001 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550771+0000", 2024-09-16T03:40:57.459 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6187000000000003e-05 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 63/31 ECSubWrite(tid=140, reqid=osd.0.0:140, at_version=63'56, trim_to=0'0, roll_forward_to=60'52))", 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.476678+0000", 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 14.930685984, 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0089615270000000004, 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.460 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476678+0000", 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476678+0000", 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476679+0000", 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.04e-06 2024-09-16T03:40:57.461 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476705+0000", 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5340000000000001e-05 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476721+0000", 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6324000000000002e-05 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.462 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476729+0000", 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5520000000000005e-06 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.477342+0000", 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00061238899999999997 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.477350+0000", 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.208e-06 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.463 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.485579+0000", 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0082287560000000003 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.485640+0000", 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0918000000000002e-05 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.464 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(osd.11.23:1 3.bs0 3:de0fee80:::smithi12138290-48:head [assert-version v28,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.479910+0000", 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 14.927453642, 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092349349999999997, 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479910+0000", 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.465 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479910+0000", 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479913+0000", 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0600000000000002e-06 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479920+0000", 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6290000000000001e-06 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479936+0000", 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5749000000000001e-05 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479945+0000", 2024-09-16T03:40:57.467 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4329999999999993e-06 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480018+0000", 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2384000000000004e-05 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480125+0000", 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107074 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.468 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.489145+0000", 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090206060000000005 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 73/31 ECSubWrite(tid=176, reqid=osd.0.0:176, at_version=73'82, trim_to=0'0, roll_forward_to=72'78))", 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.481052+0000", 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9263119529999999, 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012524485, 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.469 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481052+0000", 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481052+0000", 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.470 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481053+0000", 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9300000000000006e-07 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481077+0000", 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4111999999999999e-05 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481098+0000", 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0704e-05 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.471 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481107+0000", 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8820000000000007e-06 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481195+0000", 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7662000000000002e-05 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481204+0000", 2024-09-16T03:40:57.472 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3980000000000005e-06 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493544+0000", 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012339902 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493577+0000", 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2832e-05 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.473 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:649 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~425984] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.483387+0000", 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9239769510000002, 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.021278157999999998, 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.474 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 649 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.483387+0000", 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.483387+0000", 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.475 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.483389+0000", 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1600000000000001e-06 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.484029+0000", 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00063944299999999996 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.484041+0000", 2024-09-16T03:40:57.476 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2369e-05 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.484049+0000", 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9619999999999993e-06 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.486874+0000", 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028244390000000002 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.477 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487037+0000", 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016383700000000001 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.488288+0000", 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001250995 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493564+0000", 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052756249999999999 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.478 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504598+0000", 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011034089 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504665+0000", 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7238999999999998e-05 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.479 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:650 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~606208] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.487370+0000", 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9199946099999998, 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017507029, 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 650 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.480 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487370+0000", 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487370+0000", 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487371+0000", 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9570000000000001e-06 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.481 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487965+0000", 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00059316399999999997 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487986+0000", 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1182000000000001e-05 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487996+0000", 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0146000000000001e-05 2024-09-16T03:40:57.482 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.490298+0000", 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023018779999999998 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.490410+0000", 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000112457 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.483 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492647+0000", 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002236465 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499318+0000", 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066710809999999997 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504802+0000", 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054842770000000001 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.484 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504877+0000", 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4422e-05 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:651 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~540672] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.491437+0000", 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9159275899999999, 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013890236, 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.485 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 651 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491437+0000", 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.486 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491437+0000", 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491438+0000", 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.578e-06 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492172+0000", 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073402799999999996 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.487 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492186+0000", 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4137e-05 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492195+0000", 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2619999999999998e-06 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493674+0000", 2024-09-16T03:40:57.488 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001479803 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493779+0000", 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000104399 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.494705+0000", 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00092621299999999995 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499319+0000", 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046143490000000002 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505255+0000", 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059355179999999999 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505327+0000", 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1948999999999998e-05 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.490 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:652 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~409600] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.495027+0000", 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9123367839999998, 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010815300999999999, 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.491 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 652 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495027+0000", 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495027+0000", 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495029+0000", 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0789999999999999e-06 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495441+0000", 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000411753 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.493 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495457+0000", 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495466+0000", 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9279999999999999e-06 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495851+0000", 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038526000000000001 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496004+0000", 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015289200000000001 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496867+0000", 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00086269600000000004 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504685+0000", 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078182930000000005 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505774+0000", 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010881599999999999 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505843+0000", 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9101000000000001e-05 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:653 3.bs0 3:de0fee80:::smithi12138290-48:head [setxattr _header (57) in=7b] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:40:57.496 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9118815829999996, 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013098884, 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 653 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4600000000000003e-07 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495487+0000", 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9389999999999999e-06 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495495+0000", 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4100000000000002e-06 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495500+0000", 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6300000000000003e-06 2024-09-16T03:40:57.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499810+0000", 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043094860000000004 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499920+0000", 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000110507 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.500346+0000", 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000425735 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504687+0000", 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043411109999999999 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508462+0000", 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037750840000000002 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508581+0000", 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011913600000000001 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:654 3.bs0 3:de0fee80:::smithi12138290-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.911854527, 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016989471999999999, 2024-09-16T03:40:57.502 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 654 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.503 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3900000000000005e-07 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495512+0000", 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2259999999999999e-06 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495519+0000", 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7769999999999997e-06 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495525+0000", 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4419999999999997e-06 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.500653+0000", 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051282619999999998 2024-09-16T03:40:57.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.500758+0000", 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105141 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508620+0000", 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078621590000000005 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508703+0000", 2024-09-16T03:40:57.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2622999999999998e-05 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.512499+0000", 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003796303 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 73/31 ECSubWrite(tid=179, reqid=osd.0.0:179, at_version=73'84, trim_to=0'0, roll_forward_to=73'82))", 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.495671+0000", 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.9116929769999995, 2024-09-16T03:40:57.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0088315359999999992, 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495671+0000", 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495671+0000", 2024-09-16T03:40:57.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495672+0000", 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2799999999999995e-07 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495695+0000", 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2566e-05 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495707+0000", 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2945999999999999e-05 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495716+0000", 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2789999999999998e-06 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495797+0000", 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1537000000000002e-05 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495806+0000", 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8980000000000002e-06 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504476+0000", 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.008670301 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504503+0000", 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6180999999999999e-05 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.511 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 73/31 ECSubWrite(tid=180, reqid=osd.0.0:180, at_version=73'86, trim_to=0'0, roll_forward_to=73'82))", 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.496292+0000", 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.91107242, 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0082336990000000006, 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496292+0000", 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496292+0000", 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496292+0000", 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6899999999999998e-07 2024-09-16T03:40:57.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496311+0000", 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9326000000000001e-05 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496322+0000", 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0431999999999999e-05 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496329+0000", 2024-09-16T03:40:57.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2899999999999997e-06 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497305+0000", 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00097570800000000004 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497313+0000", 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1510000000000004e-06 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504507+0000", 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007193957 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504525+0000", 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8366e-05 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 74/31 ECSubWrite(tid=194, reqid=osd.0.0:194, at_version=74'100, trim_to=0'0, roll_forward_to=73'96))", 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.530401+0000", 2024-09-16T03:40:57.516 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.876962786, 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092564819999999999, 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530401+0000", 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530401+0000", 2024-09-16T03:40:57.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530402+0000", 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2900000000000004e-07 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530430+0000", 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7979000000000001e-05 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530441+0000", 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0855e-05 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530449+0000", 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1750000000000005e-06 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530878+0000", 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00042890400000000003 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.520 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530941+0000", 2024-09-16T03:40:57.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3035999999999998e-05 2024-09-16T03:40:57.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.520 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.520 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539613+0000", 2024-09-16T03:40:57.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086722910000000004 2024-09-16T03:40:57.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.521 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.521 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539658+0000", 2024-09-16T03:40:57.521 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4712999999999997e-05 2024-09-16T03:40:57.521 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.521 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.521 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.521 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.521 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.521 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:57.523 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:40:57.523 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:57.521+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:57.546 INFO:tasks.rados.rados.0.smithi121.stdout:288: done (5 left) 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:289: read oid 24 snap -1 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:289: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:290: rollback oid 48 current snap is 26 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 48 to 15 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:282: finishing write tid 2 to smithi12138290-30 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:285: finishing rollback tid 0 to smithi12138290-5 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:282: finishing write tid 3 to smithi12138290-30 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:282: finishing write tid 4 to smithi12138290-30 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:285: finishing rollback tid 1 to smithi12138290-5 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:285: finishing rollback tid 2 to smithi12138290-5 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 5 v 41 (ObjNum 94 snap 13 seq_num 94) dirty exists 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:282: finishing write tid 5 to smithi12138290-30 2024-09-16T03:40:57.547 INFO:tasks.rados.rados.0.smithi121.stdout:282: finishing write tid 6 to smithi12138290-30 2024-09-16T03:40:57.548 INFO:tasks.rados.rados.0.smithi121.stdout:282: oid 30 updating version 0 to 113 2024-09-16T03:40:57.548 INFO:tasks.rados.rados.0.smithi121.stdout:282: oid 30 version 113 is already newer than 109 2024-09-16T03:40:57.548 INFO:tasks.rados.rados.0.smithi121.stdout:282: oid 30 version 113 is already newer than 111 2024-09-16T03:40:57.548 INFO:tasks.rados.rados.0.smithi121.stdout:282: oid 30 version 113 is already newer than 110 2024-09-16T03:40:57.548 INFO:tasks.rados.rados.0.smithi121.stdout:282: oid 30 version 113 is already newer than 112 2024-09-16T03:40:57.548 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 30 v 113 (ObjNum 132 snap 26 seq_num 132) dirty exists 2024-09-16T03:40:57.548 INFO:tasks.rados.rados.0.smithi121.stdout:282: left oid 30 (ObjNum 132 snap 26 seq_num 132) 2024-09-16T03:40:57.548 INFO:tasks.rados.rados.0.smithi121.stdout:287: expect (ObjNum 114 snap 23 seq_num 114) 2024-09-16T03:40:57.605 INFO:tasks.rados.rados.0.smithi121.stdout:284: expect (ObjNum 101 snap 20 seq_num 101) 2024-09-16T03:40:57.623 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:57.621+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:57.647 INFO:tasks.rados.rados.0.smithi121.stdout:286: expect (ObjNum 79 snap 11 seq_num 79) 2024-09-16T03:40:57.668 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:57.669 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:57.669 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:40:57.669 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:57.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:40:57.679 INFO:tasks.rados.rados.0.smithi121.stdout:290: finishing rollback tid 0 to smithi12138290-48 2024-09-16T03:40:57.679 INFO:tasks.rados.rados.0.smithi121.stdout:290: finishing rollback tid 1 to smithi12138290-48 2024-09-16T03:40:57.679 INFO:tasks.rados.rados.0.smithi121.stdout:289: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:40:57.683 INFO:tasks.rados.rados.0.smithi121.stdout:290: finishing rollback tid 2 to smithi12138290-48 2024-09-16T03:40:57.683 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 48 v 58 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:40:57.683 INFO:tasks.rados.rados.0.smithi121.stdout:282: done (6 left) 2024-09-16T03:40:57.683 INFO:tasks.rados.rados.0.smithi121.stdout:284: done (5 left) 2024-09-16T03:40:57.683 INFO:tasks.rados.rados.0.smithi121.stdout:285: done (4 left) 2024-09-16T03:40:57.683 INFO:tasks.rados.rados.0.smithi121.stdout:286: done (3 left) 2024-09-16T03:40:57.683 INFO:tasks.rados.rados.0.smithi121.stdout:287: done (2 left) 2024-09-16T03:40:57.683 INFO:tasks.rados.rados.0.smithi121.stdout:289: done (1 left) 2024-09-16T03:40:57.683 INFO:tasks.rados.rados.0.smithi121.stdout:290: done (0 left) 2024-09-16T03:40:57.683 INFO:tasks.rados.rados.0.smithi121.stdout:291: setattr oid 25 current snap is 26 2024-09-16T03:40:57.684 INFO:tasks.rados.rados.0.smithi121.stdout:292: rollback oid 43 current snap is 26 2024-09-16T03:40:57.684 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 43 to 15 2024-09-16T03:40:57.685 INFO:tasks.rados.rados.0.smithi121.stdout:293: rmattr oid 9 current snap is 26 2024-09-16T03:40:57.685 INFO:tasks.rados.rados.0.smithi121.stdout:293: done (2 left) 2024-09-16T03:40:57.685 INFO:tasks.rados.rados.0.smithi121.stdout:294: delete oid 26 current snap is 26 2024-09-16T03:40:57.688 INFO:tasks.rados.rados.0.smithi121.stdout:292: finishing rollback tid 0 to smithi12138290-43 2024-09-16T03:40:57.689 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 25 v 57 (ObjNum 122 snap 25 seq_num 122) dirty exists 2024-09-16T03:40:57.689 INFO:tasks.rados.rados.0.smithi121.stdout:292: finishing rollback tid 1 to smithi12138290-43 2024-09-16T03:40:57.689 INFO:tasks.rados.rados.0.smithi121.stdout:292: finishing rollback tid 2 to smithi12138290-43 2024-09-16T03:40:57.689 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 43 v 128 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:40:57.692 INFO:tasks.rados.rados.0.smithi121.stdout:291: done (2 left) 2024-09-16T03:40:57.692 INFO:tasks.rados.rados.0.smithi121.stdout:292: done (1 left) 2024-09-16T03:40:57.692 INFO:tasks.rados.rados.0.smithi121.stdout:294: done (0 left) 2024-09-16T03:40:57.693 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 30 current snap is 26 2024-09-16T03:40:57.693 INFO:tasks.rados.rados.0.smithi121.stdout:295: seq_num 134 ranges {5357568=704512,6062080=475136,6537216=573440,7110656=507904} 2024-09-16T03:40:57.724 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:57.722+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:57.724 INFO:tasks.rados.rados.0.smithi121.stdout:295: writing smithi12138290-30 from 5357568 to 6062080 tid 1 2024-09-16T03:40:57.726 INFO:tasks.rados.rados.0.smithi121.stdout:295: writing smithi12138290-30 from 6062080 to 6537216 tid 2 2024-09-16T03:40:57.732 INFO:tasks.rados.rados.0.smithi121.stdout:295: writing smithi12138290-30 from 6537216 to 7110656 tid 3 2024-09-16T03:40:57.733 INFO:tasks.rados.rados.0.smithi121.stdout:295: writing smithi12138290-30 from 7110656 to 7618560 tid 4 2024-09-16T03:40:57.733 INFO:tasks.rados.rados.0.smithi121.stdout:296: rollback oid 34 current snap is 26 2024-09-16T03:40:57.733 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 34 to 23 2024-09-16T03:40:57.733 INFO:tasks.rados.rados.0.smithi121.stdout:295: finishing write tid 1 to smithi12138290-30 2024-09-16T03:40:57.734 INFO:tasks.rados.rados.0.smithi121.stdout:295: finishing write tid 2 to smithi12138290-30 2024-09-16T03:40:57.734 INFO:tasks.rados.rados.0.smithi121.stdout:297: snap_create 2024-09-16T03:40:57.736 INFO:tasks.rados.rados.0.smithi121.stdout:295: finishing write tid 3 to smithi12138290-30 2024-09-16T03:40:57.737 INFO:tasks.rados.rados.0.smithi121.stdout:296: finishing rollback tid 0 to smithi12138290-34 2024-09-16T03:40:57.738 INFO:tasks.rados.rados.0.smithi121.stdout:295: finishing write tid 4 to smithi12138290-30 2024-09-16T03:40:57.740 INFO:tasks.rados.rados.0.smithi121.stdout:296: finishing rollback tid 1 to smithi12138290-34 2024-09-16T03:40:57.741 INFO:tasks.rados.rados.0.smithi121.stdout:295: finishing write tid 5 to smithi12138290-30 2024-09-16T03:40:57.741 INFO:tasks.rados.rados.0.smithi121.stdout:296: finishing rollback tid 2 to smithi12138290-34 2024-09-16T03:40:57.742 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 34 v 52 (ObjNum 103 snap 20 seq_num 103) dirty exists 2024-09-16T03:40:57.742 INFO:tasks.rados.rados.0.smithi121.stdout:295: finishing write tid 6 to smithi12138290-30 2024-09-16T03:40:57.742 INFO:tasks.rados.rados.0.smithi121.stdout:295: oid 30 updating version 0 to 125 2024-09-16T03:40:57.742 INFO:tasks.rados.rados.0.smithi121.stdout:295: oid 30 updating version 125 to 126 2024-09-16T03:40:57.742 INFO:tasks.rados.rados.0.smithi121.stdout:295: oid 30 version 126 is already newer than 122 2024-09-16T03:40:57.742 INFO:tasks.rados.rados.0.smithi121.stdout:295: oid 30 version 126 is already newer than 123 2024-09-16T03:40:57.742 INFO:tasks.rados.rados.0.smithi121.stdout:295: oid 30 version 126 is already newer than 124 2024-09-16T03:40:57.742 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 30 v 126 (ObjNum 134 snap 26 seq_num 134) dirty exists 2024-09-16T03:40:57.742 INFO:tasks.rados.rados.0.smithi121.stdout:295: left oid 30 (ObjNum 134 snap 26 seq_num 134) 2024-09-16T03:40:57.785 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:57.785 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:40:57.785 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:40:57.786 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:40:57.786 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:57.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:40:57.825 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:40:57.824+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:40:57.898 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:40:57.898 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:40:57.898 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:40:57.898 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:40:57.898 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.898 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:205 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~655360] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:57.899 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:40:57.899 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 41.372501839999998, 2024-09-16T03:40:57.899 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075538640000000004, 2024-09-16T03:40:57.899 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.899 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.899 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.899 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.899 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 205 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.900 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523025+0000", 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3519999999999999e-06 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523819+0000", 2024-09-16T03:40:57.901 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00079422800000000001 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523834+0000", 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5333e-05 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523842+0000", 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2960000000000001e-06 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.902 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.903 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.903 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523903+0000", 2024-09-16T03:40:57.903 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1124000000000006e-05 2024-09-16T03:40:57.903 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.903 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.903 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.903 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523981+0000", 2024-09-16T03:40:57.903 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8166999999999999e-05 2024-09-16T03:40:57.903 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.904 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.904 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.904 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525035+0000", 2024-09-16T03:40:57.904 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010540969999999999 2024-09-16T03:40:57.904 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.904 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.904 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.904 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528679+0000", 2024-09-16T03:40:57.904 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036443090000000001 2024-09-16T03:40:57.904 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.905 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.905 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.905 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530521+0000", 2024-09-16T03:40:57.905 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018416789999999999 2024-09-16T03:40:57.905 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.905 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.905 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530577+0000", 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6279000000000001e-05 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:208 3.6s0 3:656a4357:::smithi12138290-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.525875+0000", 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 41.369649756000001, 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006650698, 2024-09-16T03:40:57.906 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.907 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.907 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:57.907 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.907 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.907 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.907 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 208 2024-09-16T03:40:57.907 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.907 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.907 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.908 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.908 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525875+0000", 2024-09-16T03:40:57.908 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.908 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.908 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.908 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.908 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525875+0000", 2024-09-16T03:40:57.908 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.908 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.909 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.909 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.909 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525876+0000", 2024-09-16T03:40:57.909 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:40:57.909 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.909 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.909 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.909 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525877+0000", 2024-09-16T03:40:57.909 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.477e-06 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525883+0000", 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2689999999999999e-06 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525895+0000", 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2384e-05 2024-09-16T03:40:57.910 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528574+0000", 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002678854 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528669+0000", 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4968000000000003e-05 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.911 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.912 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.531658+0000", 2024-09-16T03:40:57.912 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029888029999999999 2024-09-16T03:40:57.912 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.912 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.912 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.912 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.531711+0000", 2024-09-16T03:40:57.913 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3180999999999998e-05 2024-09-16T03:40:57.913 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.913 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.913 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.913 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.532526+0000", 2024-09-16T03:40:57.913 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00081546099999999996 2024-09-16T03:40:57.913 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.914 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.914 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.914 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.914 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.914 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:305 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:57.914 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:40:57.915 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 29.408276105999999, 2024-09-16T03:40:57.915 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087742589999999995, 2024-09-16T03:40:57.915 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.915 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.915 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.915 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.915 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.915 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.916 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 305 2024-09-16T03:40:57.916 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.916 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.916 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.916 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.916 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:40:57.916 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.917 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.917 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.917 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.917 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:40:57.917 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.917 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.917 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487251+0000", 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.824e-06 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487259+0000", 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1689999999999992e-06 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.918 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487275+0000", 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5889e-05 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487283+0000", 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9710000000000004e-06 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487385+0000", 2024-09-16T03:40:57.919 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010225099999999999 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487554+0000", 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016910199999999999 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489000+0000", 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001445499 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.920 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.921 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492361+0000", 2024-09-16T03:40:57.921 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033615799999999999 2024-09-16T03:40:57.921 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.921 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.921 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.921 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495935+0000", 2024-09-16T03:40:57.921 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0035732899999999998 2024-09-16T03:40:57.921 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.921 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.921 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.922 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496023+0000", 2024-09-16T03:40:57.922 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8684000000000007e-05 2024-09-16T03:40:57.922 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.922 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.922 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.922 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.922 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.922 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:306 3.6s0 3:64535dd7:::smithi12138290-18:head [rollback 5] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:57.922 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 29.408223243999998, 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0099232220000000006, 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 306 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.923 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.924 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487303+0000", 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.44e-07 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487306+0000", 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.817e-06 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487314+0000", 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4000000000000003e-06 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.925 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487319+0000", 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2379999999999997e-06 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489535+0000", 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002216347 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489617+0000", 2024-09-16T03:40:57.926 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1559999999999998e-05 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490002+0000", 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038484000000000001 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492363+0000", 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023609070000000002 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.927 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497154+0000", 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047910110000000004 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497225+0000", 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1557999999999995e-05 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.928 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:307 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 29.408181427999999, 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010112101, 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 307 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.929 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.51e-07 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487356+0000", 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1793999999999999e-05 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487362+0000", 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8100000000000003e-06 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487366+0000", 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8670000000000004e-06 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490302+0000", 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029359299999999998 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490396+0000", 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4134000000000004e-05 2024-09-16T03:40:57.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490929+0000", 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00053311500000000002 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492364+0000", 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014347279999999999 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497383+0000", 2024-09-16T03:40:57.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050189220000000003 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497456+0000", 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3250000000000005e-05 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:348 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498255+0000", 2024-09-16T03:40:57.934 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.397269917999999, 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069791749999999998, 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 348 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498255+0000", 2024-09-16T03:40:57.935 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498255+0000", 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498257+0000", 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4649999999999999e-06 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.936 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498264+0000", 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5440000000000001e-06 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498276+0000", 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1324e-05 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498283+0000", 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5730000000000002e-06 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498455+0000", 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000171533 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498560+0000", 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105282 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499260+0000", 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000700042 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505047+0000", 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057870489999999998 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505134+0000", 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6939000000000002e-05 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505235+0000", 2024-09-16T03:40:57.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000100424 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:349 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [rollback 3] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.397226891999999, 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084181240000000008, 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.940 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 349 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498299+0000", 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6600000000000002e-07 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498301+0000", 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3860000000000001e-06 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498307+0000", 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8590000000000001e-06 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498312+0000", 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6249999999999998e-06 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499617+0000", 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0013049660000000001 2024-09-16T03:40:57.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499672+0000", 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5787e-05 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500118+0000", 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044587499999999998 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505378+0000", 2024-09-16T03:40:57.944 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005259789 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506634+0000", 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012560169999999999 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506716+0000", 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2354000000000005e-05 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.945 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:350 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.397206820000001, 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086865829999999995, 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 350 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.947 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498319+0000", 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.39e-07 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498321+0000", 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.815e-06 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498325+0000", 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8749999999999999e-06 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.948 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498329+0000", 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.014e-06 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500398+0000", 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020690069999999999 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500458+0000", 2024-09-16T03:40:57.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9429999999999999e-05 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500725+0000", 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026708899999999998 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505380+0000", 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046551040000000002 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506933+0000", 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015530769999999999 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507005+0000", 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1833000000000002e-05 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s2 53/31 ECSubWrite(tid=40, reqid=osd.13.0:40, at_version=53'20, trim_to=0'0, roll_forward_to=52'18))", 2024-09-16T03:40:57.951 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.393939488000001, 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007493184, 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.23e-07 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501604+0000", 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8465e-05 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501615+0000", 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.06e-05 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501623+0000", 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8019999999999992e-06 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501779+0000", 2024-09-16T03:40:57.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000156093 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501788+0000", 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3840000000000003e-06 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.509033+0000", 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0072446300000000002 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.955 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.509079+0000", 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5986999999999999e-05 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s2 53/31 ECSubWrite(tid=42, reqid=client.4615.0:362, at_version=53'21, trim_to=0'0, roll_forward_to=53'20))", 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.515864+0000", 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.379661098, 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055959920000000002, 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.956 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.515864+0000", 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.515864+0000", 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.515865+0000", 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0900000000000001e-07 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516245+0000", 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038044000000000001 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516260+0000", 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4948e-05 2024-09-16T03:40:57.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516269+0000", 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9430000000000006e-06 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516348+0000", 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8817000000000001e-05 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.516357+0000", 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3500000000000003e-06 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.521430+0000", 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050732219999999996 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.521460+0000", 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9663e-05 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.960 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:412 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~737280] snapc d=[d,b,8,5,3] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.292594081000001, 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007659028, 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.961 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 412 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.962 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602932+0000", 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3499999999999995e-07 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603668+0000", 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073570899999999997 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603677+0000", 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4180000000000003e-06 2024-09-16T03:40:57.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603702+0000", 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5225999999999999e-05 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603754+0000", 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1391000000000003e-05 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603826+0000", 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2648e-05 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604587+0000", 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076084500000000001 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.608450+0000", 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038632409999999999 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.965 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610532+0000", 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020811580000000001 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610590+0000", 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8656999999999998e-05 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:413 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~573440] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.966 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.604921+0000", 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.290603820000001, 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060231779999999997, 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 413 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604921+0000", 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604921+0000", 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604922+0000", 2024-09-16T03:40:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0440000000000001e-06 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606010+0000", 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001087743 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606023+0000", 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2384e-05 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606029+0000", 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3160000000000002e-06 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606071+0000", 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1727999999999998e-05 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606140+0000", 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8895999999999999e-05 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.970 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606663+0000", 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00052336799999999997 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610599+0000", 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039365729999999996 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610895+0000", 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00029558699999999999 2024-09-16T03:40:57.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610945+0000", 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9539000000000001e-05 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:415 3.6s0 3:64535dd7:::smithi12138290-18:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.288987531, 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0056604400000000001, 2024-09-16T03:40:57.972 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 415 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606538+0000", 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.53e-07 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606540+0000", 2024-09-16T03:40:57.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0430000000000002e-06 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606545+0000", 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5059999999999997e-06 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606547+0000", 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7530000000000002e-06 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.609850+0000", 2024-09-16T03:40:57.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003302471 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.609914+0000", 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3992000000000007e-05 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610241+0000", 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000326767 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610789+0000", 2024-09-16T03:40:57.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00054870099999999996 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612138+0000", 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001348807 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612198+0000", 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0047000000000003e-05 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:416 3.6s0 3:64535dd7:::smithi12138290-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.606551+0000", 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.288973964, 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068823779999999998, 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 416 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.978 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606551+0000", 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606551+0000", 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606552+0000", 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5699999999999999e-07 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606553+0000", 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.297e-06 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606560+0000", 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2919999999999998e-06 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.606563+0000", 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0570000000000001e-06 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.611182+0000", 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046187349999999997 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.611234+0000", 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2080000000000003e-05 2024-09-16T03:40:57.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612474+0000", 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001239892 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.612554+0000", 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9999000000000004e-05 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.613434+0000", 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00087976899999999997 2024-09-16T03:40:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:466 3.6s0 3:6167ac73:::smithi12138290-7:head [stat,delete] snapc 11=[11,10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.420480734000002, 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0074852260000000002, 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.983 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 466 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:40:57.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475046+0000", 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3349999999999999e-06 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475057+0000", 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1158999999999999e-05 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475069+0000", 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1640999999999999e-05 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475076+0000", 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5379999999999997e-06 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475130+0000", 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4267000000000001e-05 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475135+0000", 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8640000000000004e-06 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478234+0000", 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030989450000000001 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478358+0000", 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000123666 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478978+0000", 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00061992000000000004 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.479700+0000", 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00072176900000000003 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.482485+0000", 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027855340000000001 2024-09-16T03:40:57.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.482530+0000", 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4588000000000001e-05 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s2 63/55 ECSubWrite(tid=66, reqid=osd.1.0:64, at_version=0'0, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.479779+0000", 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 15.415746527, 2024-09-16T03:40:57.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058770380000000002, 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479779+0000", 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479779+0000", 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479779+0000", 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479791+0000", 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1741e-05 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479802+0000", 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0873000000000001e-05 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479811+0000", 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8580000000000006e-06 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479908+0000", 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7406999999999998e-05 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479919+0000", 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0847e-05 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.485609+0000", 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0056899790000000004 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.485656+0000", 2024-09-16T03:40:57.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6743000000000003e-05 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:688 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~475136] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.539837+0000", 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.3556886190000004, 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010608545, 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:57.994 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 688 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539837+0000", 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:57.995 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539837+0000", 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539839+0000", 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2479999999999999e-06 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540778+0000", 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00093899100000000002 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540797+0000", 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9191e-05 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540806+0000", 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6200000000000005e-06 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540874+0000", 2024-09-16T03:40:57.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7949999999999998e-05 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.541002+0000", 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000128131 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542267+0000", 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012655019999999999 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:57.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550290+0000", 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080230189999999993 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550352+0000", 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2162999999999994e-05 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550445+0000", 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2730000000000002e-05 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:57.999 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:689 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~655360] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.544092+0000", 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.3514330860000001, 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081954690000000004, 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:58.000 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 689 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544092+0000", 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544092+0000", 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:58.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544094+0000", 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8619999999999999e-06 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545122+0000", 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010280389999999999 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:58.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545135+0000", 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3426e-05 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545143+0000", 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8269999999999998e-06 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545656+0000", 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00051290899999999998 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545817+0000", 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016028200000000001 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.546825+0000", 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001008253 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550450+0000", 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036248209999999999 2024-09-16T03:40:58.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.552236+0000", 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017865380000000001 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.552288+0000", 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1511999999999997e-05 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:690 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~622592] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.547745+0000", 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.3477803279999998, 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0070259609999999998, 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 690 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:58.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547745+0000", 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547745+0000", 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547747+0000", 2024-09-16T03:40:58.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.548423+0000", 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00067583499999999996 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.548441+0000", 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7597000000000001e-05 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.548460+0000", 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9099000000000001e-05 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.548556+0000", 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5903000000000007e-05 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.548674+0000", 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118746 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.549396+0000", 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000721381 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550929+0000", 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001533508 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.554724+0000", 2024-09-16T03:40:58.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037944670000000002 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.554771+0000", 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7179999999999999e-05 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:693 3.6s0 3:656a4357:::smithi12138290-23:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.753628+0000", 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.1418968950000004, 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060778899999999999, 2024-09-16T03:40:58.011 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 693 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.753628+0000", 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.753628+0000", 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.753629+0000", 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.65e-07 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.753631+0000", 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.875e-06 2024-09-16T03:40:58.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.753635+0000", 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6480000000000002e-06 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.753639+0000", 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2990000000000001e-06 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.755580+0000", 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001941159 2024-09-16T03:40:58.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.755644+0000", 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4015000000000003e-05 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.758725+0000", 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030814280000000002 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.758789+0000", 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3711e-05 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.759706+0000", 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00091739000000000002 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:40:58.016 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:40:58.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:40:58.018 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:58.016+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:58.032 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:40:58.118 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:58.117+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:58.122 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:58.122 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:58.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:40:58.146 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:40:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:40:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:40:58.146 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:40:58.155 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-09-16T03:40:58.219 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:40:58.217+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:40:58.233 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:58.233 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:58.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:40:58.320 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:40:58.319+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:40:58.346 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:58.347 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:58.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:58.347 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:58.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.347 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/31 ECSubWrite(tid=30, reqid=client.4615.0:301, at_version=49'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:58.347 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.487167+0000", 2024-09-16T03:40:58.347 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.856672301, 2024-09-16T03:40:58.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005489954, 2024-09-16T03:40:58.347 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.347 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487167+0000", 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487167+0000", 2024-09-16T03:40:58.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487167+0000", 2024-09-16T03:40:58.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4900000000000001e-07 2024-09-16T03:40:58.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487497+0000", 2024-09-16T03:40:58.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032991100000000001 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487511+0000", 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3652000000000001e-05 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487522+0000", 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1224e-05 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487609+0000", 2024-09-16T03:40:58.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6839e-05 2024-09-16T03:40:58.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487627+0000", 2024-09-16T03:40:58.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8173e-05 2024-09-16T03:40:58.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492620+0000", 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049931430000000002 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492657+0000", 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6563000000000002e-05 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.352 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=109, reqid=osd.6.0:109, at_version=73'87, trim_to=0'0, roll_forward_to=72'85))", 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.477672+0000", 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8661670170000004, 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056828829999999997, 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477672+0000", 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477672+0000", 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477672+0000", 2024-09-16T03:40:58.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4400000000000001e-07 2024-09-16T03:40:58.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477692+0000", 2024-09-16T03:40:58.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9857999999999999e-05 2024-09-16T03:40:58.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477703+0000", 2024-09-16T03:40:58.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1453e-05 2024-09-16T03:40:58.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477712+0000", 2024-09-16T03:40:58.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5390000000000007e-06 2024-09-16T03:40:58.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477795+0000", 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2634999999999997e-05 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477805+0000", 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0135e-05 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483322+0000", 2024-09-16T03:40:58.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055167840000000003 2024-09-16T03:40:58.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483355+0000", 2024-09-16T03:40:58.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3235000000000003e-05 2024-09-16T03:40:58.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 73/31 ECSubWrite(tid=83, reqid=osd.1.0:83, at_version=73'38, trim_to=0'0, roll_forward_to=72'36))", 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8659763280000004, 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059263110000000001, 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.359 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.360 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:40:58.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:40:58.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.17e-07 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477878+0000", 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4824000000000001e-05 2024-09-16T03:40:58.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477887+0000", 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6509999999999996e-06 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477895+0000", 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.401e-06 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477945+0000", 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0555e-05 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477953+0000", 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.875e-06 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483761+0000", 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058074750000000003 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483789+0000", 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8313000000000001e-05 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 73/31 ECSubWrite(tid=85, reqid=osd.1.0:85, at_version=73'40, trim_to=0'0, roll_forward_to=72'36))", 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8626796030000001, 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071150040000000003, 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481160+0000", 2024-09-16T03:40:58.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481177+0000", 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7368999999999999e-05 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481186+0000", 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1940000000000004e-06 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481194+0000", 2024-09-16T03:40:58.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4209999999999998e-06 2024-09-16T03:40:58.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481262+0000", 2024-09-16T03:40:58.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8312000000000006e-05 2024-09-16T03:40:58.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481272+0000", 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3519999999999996e-06 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488236+0000", 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069643860000000004 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488274+0000", 2024-09-16T03:40:58.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8390999999999999e-05 2024-09-16T03:40:58.370 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.370 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.370 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.370 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=111, reqid=osd.6.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:40:58.370 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8522241810000004, 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006602218, 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:40:58.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:40:58.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:40:58.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491625+0000", 2024-09-16T03:40:58.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7249999999999992e-06 2024-09-16T03:40:58.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491634+0000", 2024-09-16T03:40:58.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3370000000000006e-06 2024-09-16T03:40:58.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491642+0000", 2024-09-16T03:40:58.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0770000000000002e-06 2024-09-16T03:40:58.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491764+0000", 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012253500000000001 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491772+0000", 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1499999999999999e-06 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498186+0000", 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064136640000000003 2024-09-16T03:40:58.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498217+0000", 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1080000000000001e-05 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 73/55 ECSubWrite(tid=98, reqid=osd.10.0:98, at_version=73'24, trim_to=0'0, roll_forward_to=56'20))", 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8503582329999997, 2024-09-16T03:40:58.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065373489999999996, 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:40:58.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.61e-07 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493506+0000", 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4556999999999999e-05 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493520+0000", 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3804e-05 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493528+0000", 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9340000000000006e-06 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493606+0000", 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8144000000000003e-05 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493617+0000", 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1368e-05 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499976+0000", 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006358463 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500018+0000", 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2317999999999997e-05 2024-09-16T03:40:58.380 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.381 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.381 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=112, reqid=osd.6.0:112, at_version=73'91, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:40:58.381 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:40:58.381 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8500342359999999, 2024-09-16T03:40:58.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062537729999999998, 2024-09-16T03:40:58.381 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.382 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.382 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.382 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:40:58.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5600000000000005e-07 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493823+0000", 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7804000000000001e-05 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493835+0000", 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2197e-05 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493845+0000", 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6409999999999997e-06 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493899+0000", 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4418e-05 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493909+0000", 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5670000000000001e-06 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500026+0000", 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006117468 2024-09-16T03:40:58.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500058+0000", 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2021999999999999e-05 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=113, reqid=osd.6.0:113, at_version=73'93, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8497513479999999, 2024-09-16T03:40:58.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083691979999999996, 2024-09-16T03:40:58.387 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.387 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.387 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.387 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:40:58.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0399999999999995e-07 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494110+0000", 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1750999999999999e-05 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494122+0000", 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1532000000000001e-05 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494129+0000", 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7300000000000005e-06 2024-09-16T03:40:58.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495262+0000", 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011327679999999999 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495271+0000", 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7739999999999993e-06 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502425+0000", 2024-09-16T03:40:58.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071538160000000003 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502457+0000", 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2122999999999997e-05 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 73/55 ECSubWrite(tid=103, reqid=osd.10.0:98, at_version=0'0, trim_to=0'0, roll_forward_to=73'24))", 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502502+0000", 2024-09-16T03:40:58.391 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.841337416, 2024-09-16T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005602788, 2024-09-16T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502502+0000", 2024-09-16T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502502+0000", 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502502+0000", 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.92e-07 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502511+0000", 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1330000000000005e-06 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502533+0000", 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1820000000000001e-05 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502550+0000", 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6915e-05 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502610+0000", 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0050999999999998e-05 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502620+0000", 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7210000000000006e-06 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508066+0000", 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005445943 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508104+0000", 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8513000000000002e-05 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.396 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:682 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.8162587800000001, 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095978390000000004, 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 682 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527582+0000", 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7910000000000001e-06 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527590+0000", 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5320000000000001e-06 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527608+0000", 2024-09-16T03:40:58.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8496999999999999e-05 2024-09-16T03:40:58.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527622+0000", 2024-09-16T03:40:58.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3583e-05 2024-09-16T03:40:58.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527699+0000", 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7799000000000005e-05 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527707+0000", 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.530874+0000", 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031660529999999998 2024-09-16T03:40:58.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:58.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.530988+0000", 2024-09-16T03:40:58.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114376 2024-09-16T03:40:58.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.531781+0000", 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079327 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536093+0000", 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043113819999999999 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.537011+0000", 2024-09-16T03:40:58.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091858900000000004 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.537178+0000", 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016688799999999999 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:683 3.es0 3:71ce6426:::smithi12138290-50:head [rollback 18] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.816184067, 2024-09-16T03:40:58.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015112265, 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 683 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527659+0000", 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4510000000000001e-06 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527667+0000", 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527673+0000", 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0100000000000001e-06 2024-09-16T03:40:58.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527773+0000", 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100029 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527780+0000", 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3239999999999999e-06 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.532223+0000", 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044434870000000003 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.532315+0000", 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2255000000000004e-05 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533195+0000", 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087904200000000002 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536094+0000", 2024-09-16T03:40:58.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028993840000000001 2024-09-16T03:40:58.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:58.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542627+0000", 2024-09-16T03:40:58.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006533021 2024-09-16T03:40:58.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542767+0000", 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014024 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:684 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.81615503, 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015421104, 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:58.412 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 684 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000005e-07 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527687+0000", 2024-09-16T03:40:58.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4609999999999998e-06 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527696+0000", 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7600000000000008e-06 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527701+0000", 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3789999999999997e-06 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527803+0000", 2024-09-16T03:40:58.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010234900000000001 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527809+0000", 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2870000000000004e-06 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533474+0000", 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056649090000000001 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:40:58.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533571+0000", 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7552000000000004e-05 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.534004+0000", 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00043247700000000001 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536095+0000", 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020918460000000001 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542953+0000", 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068572249999999998 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.543105+0000", 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015232000000000001 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=138, reqid=osd.6.0:138, at_version=78'117, trim_to=0'0, roll_forward_to=77'113))", 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.796579272, 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064033809999999997, 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547261+0000", 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.032e-06 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547287+0000", 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5905e-05 2024-09-16T03:40:58.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547301+0000", 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3905e-05 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547310+0000", 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4590000000000004e-06 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547537+0000", 2024-09-16T03:40:58.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022751299999999999 2024-09-16T03:40:58.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547553+0000", 2024-09-16T03:40:58.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5113000000000001e-05 2024-09-16T03:40:58.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553619+0000", 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060667450000000001 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553663+0000", 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3708999999999998e-05 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 78/55 ECSubWrite(tid=122, reqid=osd.10.0:122, at_version=78'32, trim_to=0'0, roll_forward_to=76'30))", 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.79380215200000004, 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006109504, 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:40:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9100000000000004e-07 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550065+0000", 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7671999999999999e-05 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550075+0000", 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5540000000000003e-06 2024-09-16T03:40:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550083+0000", 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8119999999999997e-06 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550154+0000", 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0167999999999996e-05 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550163+0000", 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7850000000000003e-06 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556099+0000", 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059356310000000002 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556146+0000", 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7391000000000001e-05 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=139, reqid=osd.6.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.78973682300000003, 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078423550000000005, 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554103+0000", 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7700000000000004e-07 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554111+0000", 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6119999999999999e-06 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554122+0000", 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0587999999999999e-05 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554129+0000", 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554193+0000", 2024-09-16T03:40:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3522000000000006e-05 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554204+0000", 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.083e-05 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561904+0000", 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077002540000000001 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561944+0000", 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0538999999999999e-05 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.as3 78/55 ECSubWrite(tid=161, reqid=osd.12.0:161, at_version=78'43, trim_to=0'0, roll_forward_to=77'41))", 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:40:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.78901660399999995, 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067129479999999998, 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554823+0000", 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554849+0000", 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6461999999999998e-05 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554862+0000", 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2918000000000001e-05 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554872+0000", 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.005e-05 2024-09-16T03:40:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554971+0000", 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8665999999999994e-05 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554981+0000", 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0161e-05 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561484+0000", 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006503404 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561535+0000", 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0934999999999999e-05 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 78/31 ECSubWrite(tid=133, reqid=osd.1.0:133, at_version=78'56, trim_to=0'0, roll_forward_to=76'52))", 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.78851324199999995, 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006360844, 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555327+0000", 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1299999999999998e-07 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555347+0000", 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0389000000000001e-05 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555359+0000", 2024-09-16T03:40:58.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2371999999999999e-05 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555368+0000", 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0480000000000004e-06 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555454+0000", 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5662999999999996e-05 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555464+0000", 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0018e-05 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561663+0000", 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061989549999999999 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561687+0000", 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3485999999999998e-05 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.as3 78/55 ECSubWrite(tid=163, reqid=osd.12.0:163, at_version=78'45, trim_to=0'0, roll_forward_to=77'41))", 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.78763331000000003, 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078281010000000005, 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.445 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556207+0000", 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0809999999999999e-06 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556231+0000", 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.406e-05 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556244+0000", 2024-09-16T03:40:58.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3696999999999999e-05 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556258+0000", 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3342e-05 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556856+0000", 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059772700000000005 2024-09-16T03:40:58.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556865+0000", 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.166e-06 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564009+0000", 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007144358 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564034+0000", 2024-09-16T03:40:58.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4669999999999999e-05 2024-09-16T03:40:58.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.450 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.450 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=140, reqid=osd.6.0:140, at_version=78'119, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:40:58.450 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:40:58.450 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.78622408499999996, 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064168259999999996, 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557616+0000", 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9899999999999999e-07 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557640+0000", 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4034000000000001e-05 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557651+0000", 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1382999999999999e-05 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557660+0000", 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557721+0000", 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1484999999999998e-05 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557731+0000", 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0051000000000001e-05 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.563993+0000", 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062610649999999997 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564032+0000", 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9193999999999998e-05 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=141, reqid=osd.6.0:141, at_version=78'121, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.558959+0000", 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.78487981200000001, 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054373490000000002, 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.558959+0000", 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.558959+0000", 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.558960+0000", 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8000000000000005e-07 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.558981+0000", 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0854000000000001e-05 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.558991+0000", 2024-09-16T03:40:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7890000000000006e-06 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.558998+0000", 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8639999999999996e-06 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559233+0000", 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023478 2024-09-16T03:40:58.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559242+0000", 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9789999999999999e-06 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564359+0000", 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051171610000000003 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564396+0000", 2024-09-16T03:40:58.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7141999999999998e-05 2024-09-16T03:40:58.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.461 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.461 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.461 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:58.463 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:40:58.461+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:40:58.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:40:58.549 INFO:tasks.rados.rados.0.smithi121.stdout:295: done (2 left) 2024-09-16T03:40:58.549 INFO:tasks.rados.rados.0.smithi121.stdout:296: done (1 left) 2024-09-16T03:40:58.549 INFO:tasks.rados.rados.0.smithi121.stdout:297: done (0 left) 2024-09-16T03:40:58.549 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 5 current snap is 27 2024-09-16T03:40:58.549 INFO:tasks.rados.rados.0.smithi121.stdout:298: seq_num 135 ranges {1589248=720896,2310144=622592,2932736=688128,3620864=245760} 2024-09-16T03:40:58.558 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:40:58.561 INFO:tasks.rados.rados.0.smithi121.stdout:298: writing smithi12138290-5 from 1589248 to 2310144 tid 1 2024-09-16T03:40:58.563 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:58.562+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:58.565 INFO:tasks.rados.rados.0.smithi121.stdout:298: writing smithi12138290-5 from 2310144 to 2932736 tid 2 2024-09-16T03:40:58.569 INFO:tasks.rados.rados.0.smithi121.stdout:298: writing smithi12138290-5 from 2932736 to 3620864 tid 3 2024-09-16T03:40:58.569 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:58.569 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:58.570 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:40:58.570 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:58.571 INFO:tasks.rados.rados.0.smithi121.stdout:298: writing smithi12138290-5 from 3620864 to 3866624 tid 4 2024-09-16T03:40:58.571 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 42 current snap is 27 2024-09-16T03:40:58.571 INFO:tasks.rados.rados.0.smithi121.stdout:298: finishing write tid 1 to smithi12138290-5 2024-09-16T03:40:58.571 INFO:tasks.rados.rados.0.smithi121.stdout:299: seq_num 136 ranges {2260992=737280,2998272=770048,3768320=540672,4308992=245760} 2024-09-16T03:40:58.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:40:58.587 INFO:tasks.rados.rados.0.smithi121.stdout:299: writing smithi12138290-42 from 2260992 to 2998272 tid 1 2024-09-16T03:40:58.592 INFO:tasks.rados.rados.0.smithi121.stdout:299: writing smithi12138290-42 from 2998272 to 3768320 tid 2 2024-09-16T03:40:58.595 INFO:tasks.rados.rados.0.smithi121.stdout:299: writing smithi12138290-42 from 3768320 to 4308992 tid 3 2024-09-16T03:40:58.597 INFO:tasks.rados.rados.0.smithi121.stdout:299: writing smithi12138290-42 from 4308992 to 4554752 tid 4 2024-09-16T03:40:58.597 INFO:tasks.rados.rados.0.smithi121.stdout:298: finishing write tid 2 to smithi12138290-5 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:298: finishing write tid 3 to smithi12138290-5 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:298: finishing write tid 4 to smithi12138290-5 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:300: read oid 10 snap -1 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:300: expect (ObjNum 61 snap 2 seq_num 61) 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:301: rollback oid 8 current snap is 27 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 8 to 15 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:298: finishing write tid 5 to smithi12138290-5 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:298: finishing write tid 6 to smithi12138290-5 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:298: oid 5 updating version 0 to 48 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:298: oid 5 updating version 48 to 51 2024-09-16T03:40:58.598 INFO:tasks.rados.rados.0.smithi121.stdout:298: oid 5 version 51 is already newer than 49 2024-09-16T03:40:58.599 INFO:tasks.rados.rados.0.smithi121.stdout:298: oid 5 version 51 is already newer than 50 2024-09-16T03:40:58.599 INFO:tasks.rados.rados.0.smithi121.stdout:298: oid 5 version 51 is already newer than 47 2024-09-16T03:40:58.599 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 5 v 51 (ObjNum 135 snap 27 seq_num 135) dirty exists 2024-09-16T03:40:58.599 INFO:tasks.rados.rados.0.smithi121.stdout:298: left oid 5 (ObjNum 135 snap 27 seq_num 135) 2024-09-16T03:40:58.599 INFO:tasks.rados.rados.0.smithi121.stdout:299: finishing write tid 1 to smithi12138290-42 2024-09-16T03:40:58.599 INFO:tasks.rados.rados.0.smithi121.stdout:299: finishing write tid 2 to smithi12138290-42 2024-09-16T03:40:58.599 INFO:tasks.rados.rados.0.smithi121.stdout:298: done (3 left) 2024-09-16T03:40:58.599 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 24 current snap is 27 2024-09-16T03:40:58.599 INFO:tasks.rados.rados.0.smithi121.stdout:302: seq_num 137 ranges {131072=753664,884736=786432,1671168=655360,2326528=114688} 2024-09-16T03:40:58.602 INFO:tasks.rados.rados.0.smithi121.stdout:302: writing smithi12138290-24 from 131072 to 884736 tid 1 2024-09-16T03:40:58.606 INFO:tasks.rados.rados.0.smithi121.stdout:302: writing smithi12138290-24 from 884736 to 1671168 tid 2 2024-09-16T03:40:58.612 INFO:tasks.rados.rados.0.smithi121.stdout:302: writing smithi12138290-24 from 1671168 to 2326528 tid 3 2024-09-16T03:40:58.612 INFO:tasks.rados.rados.0.smithi121.stdout:302: writing smithi12138290-24 from 2326528 to 2441216 tid 4 2024-09-16T03:40:58.612 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 26 current snap is 27 2024-09-16T03:40:58.613 INFO:tasks.rados.rados.0.smithi121.stdout:299: finishing write tid 3 to smithi12138290-42 2024-09-16T03:40:58.613 INFO:tasks.rados.rados.0.smithi121.stdout:303: seq_num 138 ranges {0=770048,770048=540672,1310720=753664,2064384=262144} 2024-09-16T03:40:58.616 INFO:tasks.rados.rados.0.smithi121.stdout:303: writing smithi12138290-26 from 0 to 770048 tid 1 2024-09-16T03:40:58.619 INFO:tasks.rados.rados.0.smithi121.stdout:303: writing smithi12138290-26 from 770048 to 1310720 tid 2 2024-09-16T03:40:58.623 INFO:tasks.rados.rados.0.smithi121.stdout:303: writing smithi12138290-26 from 1310720 to 2064384 tid 3 2024-09-16T03:40:58.625 INFO:tasks.rados.rados.0.smithi121.stdout:303: writing smithi12138290-26 from 2064384 to 2326528 tid 4 2024-09-16T03:40:58.625 INFO:tasks.rados.rados.0.smithi121.stdout:299: finishing write tid 4 to smithi12138290-42 2024-09-16T03:40:58.625 INFO:tasks.rados.rados.0.smithi121.stdout:304: snap_remove snap 16 2024-09-16T03:40:58.664 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:40:58.663+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:40:58.674 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:58.694 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:40:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:40:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:40:58.695 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:58.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:40:58.787 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:58.786+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:58.798 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:40:58.798 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:58.816 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:40:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:58.832 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:40:58.832 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:58.848 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:40:58.848 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:40:58.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:40:58.848 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:40:58.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.848 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 42/31 ECSubWrite(tid=32, reqid=osd.14.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:58.848 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:58.848 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.281946810999997, 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075866950000000001, 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:40:58.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563726+0000", 2024-09-16T03:40:58.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.09e-07 2024-09-16T03:40:58.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563741+0000", 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5123000000000001e-05 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563761+0000", 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9335999999999999e-05 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563770+0000", 2024-09-16T03:40:58.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3770000000000002e-06 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563840+0000", 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9725000000000001e-05 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563851+0000", 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1107000000000001e-05 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571277+0000", 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074264830000000002 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571312+0000", 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4635000000000003e-05 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=30, reqid=client.4615.0:301, at_version=49'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:58.853 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.358557741999999, 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083935009999999994, 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487115+0000", 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.36e-07 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487566+0000", 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045117600000000002 2024-09-16T03:40:58.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487581+0000", 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5092000000000001e-05 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487591+0000", 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.058e-05 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487677+0000", 2024-09-16T03:40:58.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5635000000000003e-05 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487685+0000", 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2830000000000001e-06 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495470+0000", 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077842739999999999 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495508+0000", 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8225000000000001e-05 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 49/31 ECSubWrite(tid=34, reqid=client.4615.0:305, at_version=49'22, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.489031+0000", 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.356640785, 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064300850000000003, 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.858 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489031+0000", 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489031+0000", 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489032+0000", 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4700000000000002e-07 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489060+0000", 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000002e-05 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489076+0000", 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6013000000000001e-05 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489086+0000", 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9499999999999996e-06 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489306+0000", 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022061699999999999 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489317+0000", 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0895e-05 2024-09-16T03:40:58.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495421+0000", 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061040540000000003 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495461+0000", 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0139000000000003e-05 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=31, reqid=osd.6.0:31, at_version=49'22, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.489235+0000", 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.356436680000002, 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0063054690000000002, 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489235+0000", 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489235+0000", 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489236+0000", 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5499999999999996e-07 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489261+0000", 2024-09-16T03:40:58.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5045000000000002e-05 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489275+0000", 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3743e-05 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.489283+0000", 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2109999999999998e-06 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490215+0000", 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093184699999999997 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490227+0000", 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1737e-05 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495519+0000", 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052920049999999998 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495541+0000", 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1926e-05 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 49/31 ECSubWrite(tid=35, reqid=client.4615.0:306, at_version=49'23, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.355355354, 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065860449999999996, 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.867 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490318+0000", 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5600000000000004e-07 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490342+0000", 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4006e-05 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490356+0000", 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4449e-05 2024-09-16T03:40:58.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490366+0000", 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8689999999999999e-06 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494506+0000", 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041400029999999997 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494516+0000", 2024-09-16T03:40:58.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0229e-05 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496868+0000", 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023520899999999998 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496903+0000", 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4443000000000002e-05 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=32, reqid=client.4615.0:302, at_version=49'23, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.491000+0000", 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.354671906, 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065612769999999999, 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491000+0000", 2024-09-16T03:40:58.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491000+0000", 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.491001+0000", 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4600000000000003e-07 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492933+0000", 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019313749999999999 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492951+0000", 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8521000000000001e-05 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.492960+0000", 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4089999999999992e-06 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494238+0000", 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012774049999999999 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494247+0000", 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0839999999999998e-06 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497522+0000", 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032754910000000002 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.497562+0000", 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9045999999999999e-05 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 73/31 ECSubWrite(tid=56, reqid=osd.8.0:56, at_version=73'42, trim_to=0'0, roll_forward_to=63'40))", 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.479531+0000", 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3661412219999995, 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083957719999999993, 2024-09-16T03:40:58.876 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479531+0000", 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479531+0000", 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479531+0000", 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2000000000000001e-07 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479556+0000", 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4822000000000001e-05 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479566+0000", 2024-09-16T03:40:58.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7650000000000005e-06 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479574+0000", 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7789999999999996e-06 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480246+0000", 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00067257600000000003 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480254+0000", 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3520000000000007e-06 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.487884+0000", 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076291450000000004 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.487927+0000", 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3112999999999999e-05 2024-09-16T03:40:58.880 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 73/31 ECSubWrite(tid=57, reqid=osd.8.0:57, at_version=73'44, trim_to=0'0, roll_forward_to=63'40))", 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.480913+0000", 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3647588490000002, 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087512579999999996, 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480913+0000", 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480913+0000", 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480914+0000", 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0699999999999996e-07 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480938+0000", 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3949000000000001e-05 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480949+0000", 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0984e-05 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480957+0000", 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-09-16T03:40:58.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481866+0000", 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090959899999999998 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481875+0000", 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9709999999999993e-06 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.489628+0000", 2024-09-16T03:40:58.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077526219999999998 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.489665+0000", 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6786000000000003e-05 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/31 ECSubWrite(tid=110, reqid=osd.6.0:110, at_version=73'89, trim_to=0'0, roll_forward_to=72'85))", 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.481009+0000", 2024-09-16T03:40:58.885 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3646630399999999, 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0079369019999999992, 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481009+0000", 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481009+0000", 2024-09-16T03:40:58.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481010+0000", 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8100000000000001e-07 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481030+0000", 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9636999999999999e-05 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481041+0000", 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1610999999999999e-05 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481049+0000", 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1429999999999998e-06 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481115+0000", 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5438999999999995e-05 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481125+0000", 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488908+0000", 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007783023 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488946+0000", 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8247000000000002e-05 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.889 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=91, reqid=osd.14.0:91, at_version=73'73, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.484293+0000", 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3613789069999997, 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012749313, 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484293+0000", 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484293+0000", 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484302+0000", 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3580000000000003e-06 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484318+0000", 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6327000000000001e-05 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484329+0000", 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0912e-05 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484336+0000", 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6320000000000008e-06 2024-09-16T03:40:58.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484387+0000", 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0361000000000001e-05 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484396+0000", 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5279999999999992e-06 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497001+0000", 2024-09-16T03:40:58.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012604529999999999 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497043+0000", 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1665000000000002e-05 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.361320289, 2024-09-16T03:40:58.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01525908, 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.22e-07 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484387+0000", 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4653999999999997e-05 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484394+0000", 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9550000000000003e-06 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484400+0000", 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7570000000000001e-06 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488364+0000", 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039642260000000004 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488373+0000", 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5880000000000004e-06 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499570+0000", 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011197087999999999 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499611+0000", 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.049e-05 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.898 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 73/55 ECSubWrite(tid=177, reqid=osd.0.0:174, at_version=0'0, trim_to=0'0, roll_forward_to=73'30))", 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.488711+0000", 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3569614449999996, 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010894513, 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488711+0000", 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488711+0000", 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488712+0000", 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6099999999999999e-07 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488726+0000", 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4591e-05 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488818+0000", 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1811999999999994e-05 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488829+0000", 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1114999999999999e-05 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488913+0000", 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3481999999999998e-05 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488922+0000", 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4069999999999999e-06 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499556+0000", 2024-09-16T03:40:58.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010634403000000001 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499605+0000", 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8841999999999998e-05 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/31 ECSubWrite(tid=111, reqid=osd.6.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.490976+0000", 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3546963009999997, 2024-09-16T03:40:58.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089424069999999994, 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490976+0000", 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490976+0000", 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490977+0000", 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9199999999999999e-07 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490988+0000", 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1206000000000001e-05 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491004+0000", 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6351999999999999e-05 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491013+0000", 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3489999999999992e-06 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492712+0000", 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016998219999999999 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492722+0000", 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3109999999999995e-06 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499886+0000", 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071638120000000003 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499918+0000", 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2663000000000002e-05 2024-09-16T03:40:58.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 73/31 ECSubWrite(tid=58, reqid=osd.8.0:57, at_version=0'0, trim_to=0'0, roll_forward_to=73'44))", 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.492152+0000", 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3535196630000002, 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074894860000000001, 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492152+0000", 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492152+0000", 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492153+0000", 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5899999999999995e-07 2024-09-16T03:40:58.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492165+0000", 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1952e-05 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492178+0000", 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2802e-05 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492187+0000", 2024-09-16T03:40:58.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5459999999999992e-06 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492245+0000", 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8372000000000003e-05 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492256+0000", 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0759e-05 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499611+0000", 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073555089999999997 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499642+0000", 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0686999999999999e-05 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.912 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/31 ECSubWrite(tid=112, reqid=osd.6.0:112, at_version=73'91, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493729+0000", 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3519434390000002, 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085756970000000002, 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493729+0000", 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493729+0000", 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493729+0000", 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4600000000000004e-07 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493757+0000", 2024-09-16T03:40:58.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7419999999999998e-05 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493770+0000", 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3534e-05 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493779+0000", 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3739999999999998e-06 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493870+0000", 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1544000000000003e-05 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493883+0000", 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2805e-05 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502272+0000", 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083892529999999993 2024-09-16T03:40:58.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502304+0000", 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2020999999999997e-05 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/31 ECSubWrite(tid=113, reqid=osd.6.0:113, at_version=73'93, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.494706+0000", 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3509658570000003, 2024-09-16T03:40:58.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013014379, 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494706+0000", 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494706+0000", 2024-09-16T03:40:58.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494707+0000", 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0500000000000003e-07 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494728+0000", 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1047999999999999e-05 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494742+0000", 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3672000000000001e-05 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494758+0000", 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6184999999999998e-05 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497457+0000", 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002698729 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497466+0000", 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.507681+0000", 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010215284999999999 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.507721+0000", 2024-09-16T03:40:58.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9715000000000002e-05 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=93, reqid=osd.14.0:92, at_version=0'0, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.500094+0000", 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3455785819999999, 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080041739999999993, 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500094+0000", 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500094+0000", 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500095+0000", 2024-09-16T03:40:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.031e-06 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500107+0000", 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2281e-05 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500118+0000", 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1294999999999999e-05 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500127+0000", 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5350000000000004e-06 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500189+0000", 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2278000000000002e-05 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500198+0000", 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1009999999999998e-06 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508066+0000", 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078676249999999996 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508098+0000", 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2027999999999999e-05 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=94, reqid=osd.14.0:94, at_version=73'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.501070+0000", 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3446021120000005, 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007047439, 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501070+0000", 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501070+0000", 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501071+0000", 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501097+0000", 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6899000000000001e-05 2024-09-16T03:40:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501107+0000", 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6520000000000001e-06 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501114+0000", 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9040000000000003e-06 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501169+0000", 2024-09-16T03:40:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5384999999999999e-05 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501179+0000", 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2189999999999993e-06 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508101+0000", 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069222160000000001 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508117+0000", 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6623999999999999e-05 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=95, reqid=osd.14.0:95, at_version=73'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.3436085609999999, 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086042050000000002, 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0200000000000001e-07 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502084+0000", 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9769e-05 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502098+0000", 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3365e-05 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502105+0000", 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7430000000000002e-06 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502860+0000", 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075463899999999998 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502869+0000", 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8619999999999992e-06 2024-09-16T03:40:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510637+0000", 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077679530000000002 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510668+0000", 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1072000000000003e-05 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 78/31 ECSubWrite(tid=119, reqid=osd.14.0:119, at_version=78'103, trim_to=0'0, roll_forward_to=77'101))", 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.552648+0000", 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.2930239720000001, 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0066204819999999996, 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552648+0000", 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552648+0000", 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552648+0000", 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7300000000000002e-07 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552685+0000", 2024-09-16T03:40:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6899e-05 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552699+0000", 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3518e-05 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552709+0000", 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8970000000000003e-06 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552791+0000", 2024-09-16T03:40:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1874999999999998e-05 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552797+0000", 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7610000000000002e-06 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559231+0000", 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064338989999999999 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:40:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559269+0000", 2024-09-16T03:40:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7360000000000001e-05 2024-09-16T03:40:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:40:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:40:58.940 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:40:58.941 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:40:58.941 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:58.942 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:40:58.942 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:58.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:40:58.943 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:58.939+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:58.943 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:40:58.944 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:58.944 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:40:58.944 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:58.944 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:40:58.945 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:58.945 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:40:58.945 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:58.947 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:40:58.947 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:58.964 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:40:58.964 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:58.981 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:40:58.981 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:58.996 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:40:58.997 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:59.013 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:40:59.013 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:59.030 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:40:59.030 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:59.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:40:59.030+0000 7f7da8857640 0 -- 172.21.15.76:0/643651865 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f7da4220d10 msgr2=0x7f7da4223100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:40:59.042 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:40:59.039+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:40:59.086 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:59.086 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:59.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:40:59.142 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:59.141+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:59.198 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:59.198 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:59.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:40:59.244 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:40:59.242+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:40:59.248 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:40:59.248 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:59.344 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:40:59.343+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:40:59.349 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:59.349 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:59.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:40:59.445 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:40:59.444+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:40:59.501 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:59.501 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:59.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:40:59.546 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:40:59.545+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:40:59.560 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T03:40:59.569 INFO:tasks.rados.rados.0.smithi121.stdout:301: finishing rollback tid 0 to smithi12138290-8 2024-09-16T03:40:59.569 INFO:tasks.rados.rados.0.smithi121.stdout:299: finishing write tid 5 to smithi12138290-42 2024-09-16T03:40:59.569 INFO:tasks.rados.rados.0.smithi121.stdout:301: finishing rollback tid 1 to smithi12138290-8 2024-09-16T03:40:59.569 INFO:tasks.rados.rados.0.smithi121.stdout:301: finishing rollback tid 2 to smithi12138290-8 2024-09-16T03:40:59.569 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 8 v 113 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:40:59.569 INFO:tasks.rados.rados.0.smithi121.stdout:300: expect (ObjNum 61 snap 2 seq_num 61) 2024-09-16T03:40:59.620 INFO:tasks.rados.rados.0.smithi121.stdout:299: finishing write tid 6 to smithi12138290-42 2024-09-16T03:40:59.620 INFO:tasks.rados.rados.0.smithi121.stdout:299: oid 42 updating version 0 to 62 2024-09-16T03:40:59.620 INFO:tasks.rados.rados.0.smithi121.stdout:299: oid 42 version 62 is already newer than 58 2024-09-16T03:40:59.620 INFO:tasks.rados.rados.0.smithi121.stdout:299: oid 42 version 62 is already newer than 60 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:299: oid 42 version 62 is already newer than 61 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:299: oid 42 version 62 is already newer than 59 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 42 v 62 (ObjNum 136 snap 27 seq_num 136) dirty exists 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:299: left oid 42 (ObjNum 136 snap 27 seq_num 136) 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:302: finishing write tid 1 to smithi12138290-24 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:299: done (5 left) 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:300: done (4 left) 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:301: done (3 left) 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:304: done (2 left) 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:305: read oid 20 snap -1 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:305: expect (ObjNum 119 snap 24 seq_num 119) 2024-09-16T03:40:59.621 INFO:tasks.rados.rados.0.smithi121.stdout:306: snap_remove snap 26 2024-09-16T03:40:59.648 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:40:59.646+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:40:59.666 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:59.666 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:59.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:40:59.748 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:40:59.747+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:40:59.817 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:59.817 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:59.825 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:40:59.850 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:40:59.848+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:40:59.939 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:59.939 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:59.948 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:40:59.951 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:40:59.948+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:00.051 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:00.051 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:00.051 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:41:00.048+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:41:00.059 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:41:00.152 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:00.151+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:00.163 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:00.164 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:00.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:41:00.253 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:00.250+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:00.278 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:00.278 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:00.287 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:41:00.354 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:00.351+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:00.390 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:00.390 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:00.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:41:00.455 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:00.454+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:00.502 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:00.502 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:00.511 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:41:00.556 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:00.554+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:01.438 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:41:01.441 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:01.441 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:01.441 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:41:01.442 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:01.438+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:01.444 INFO:tasks.rados.rados.0.smithi121.stdout:306: done (3 left) 2024-09-16T03:41:01.445 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 32 current snap is 27 2024-09-16T03:41:01.445 INFO:tasks.rados.rados.0.smithi121.stdout:302: finishing write tid 2 to smithi12138290-24 2024-09-16T03:41:01.445 INFO:tasks.rados.rados.0.smithi121.stdout:307: seq_num 139 ranges {1916928=786432,2703360=688128,3391488=475136,3866624=393216} 2024-09-16T03:41:01.445 INFO:tasks.rados.rados.0.smithi121.stdout:307: writing smithi12138290-32 from 1916928 to 2703360 tid 1 2024-09-16T03:41:01.445 INFO:tasks.rados.rados.0.smithi121.stdout:307: writing smithi12138290-32 from 2703360 to 3391488 tid 2 2024-09-16T03:41:01.445 INFO:tasks.rados.rados.0.smithi121.stdout:307: writing smithi12138290-32 from 3391488 to 3866624 tid 3 2024-09-16T03:41:01.445 INFO:tasks.rados.rados.0.smithi121.stdout:307: writing smithi12138290-32 from 3866624 to 4259840 tid 4 2024-09-16T03:41:01.445 INFO:tasks.rados.rados.0.smithi121.stdout:308: read oid 10 snap 14 2024-09-16T03:41:01.445 INFO:tasks.rados.rados.0.smithi121.stdout:308: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:41:01.445 INFO:tasks.rados.rados.0.smithi121.stdout:309: read oid 37 snap -1 2024-09-16T03:41:01.446 INFO:tasks.rados.rados.0.smithi121.stdout:309: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-16T03:41:01.446 INFO:tasks.rados.rados.0.smithi121.stdout:310: read oid 44 snap -1 2024-09-16T03:41:01.446 INFO:tasks.rados.rados.0.smithi121.stdout:310: expect deleted 2024-09-16T03:41:01.446 INFO:tasks.rados.rados.0.smithi121.stdout:311: rollback oid 4 current snap is 27 2024-09-16T03:41:01.446 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 4 to 24 2024-09-16T03:41:01.446 INFO:tasks.rados.rados.0.smithi121.stdout:302: finishing write tid 3 to smithi12138290-24 2024-09-16T03:41:01.446 INFO:tasks.rados.rados.0.smithi121.stdout:302: finishing write tid 4 to smithi12138290-24 2024-09-16T03:41:01.446 INFO:tasks.rados.rados.0.smithi121.stdout:302: finishing write tid 5 to smithi12138290-24 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:303: finishing write tid 1 to smithi12138290-26 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:302: finishing write tid 6 to smithi12138290-24 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:302: oid 24 updating version 0 to 50 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:302: oid 24 updating version 50 to 51 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:302: oid 24 updating version 51 to 53 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:302: oid 24 version 53 is already newer than 52 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:302: oid 24 version 53 is already newer than 49 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 24 v 53 (ObjNum 137 snap 27 seq_num 137) dirty exists 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:302: left oid 24 (ObjNum 137 snap 27 seq_num 137) 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:303: finishing write tid 2 to smithi12138290-26 2024-09-16T03:41:01.447 INFO:tasks.rados.rados.0.smithi121.stdout:303: finishing write tid 3 to smithi12138290-26 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:303: finishing write tid 4 to smithi12138290-26 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:303: finishing write tid 5 to smithi12138290-26 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:303: finishing write tid 6 to smithi12138290-26 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:303: oid 26 updating version 0 to 134 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:303: oid 26 version 134 is already newer than 133 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:303: oid 26 version 134 is already newer than 132 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:303: oid 26 version 134 is already newer than 131 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:303: oid 26 updating version 134 to 135 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 26 v 135 (ObjNum 138 snap 27 seq_num 138) dirty exists 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:303: left oid 26 (ObjNum 138 snap 27 seq_num 138) 2024-09-16T03:41:01.448 INFO:tasks.rados.rados.0.smithi121.stdout:305: expect (ObjNum 119 snap 24 seq_num 119) 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:302: done (7 left) 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:303: done (6 left) 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:305: done (5 left) 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 19 current snap is 27 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:312: seq_num 140 ranges {311296=802816,1114112=425984,1540096=720896,2260992=409600} 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:312: writing smithi12138290-19 from 311296 to 1114112 tid 1 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:312: writing smithi12138290-19 from 1114112 to 1540096 tid 2 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:312: writing smithi12138290-19 from 1540096 to 2260992 tid 3 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:312: writing smithi12138290-19 from 2260992 to 2670592 tid 4 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:313: rollback oid 16 current snap is 27 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 16 to 23 2024-09-16T03:41:01.449 INFO:tasks.rados.rados.0.smithi121.stdout:308: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:41:01.450 INFO:tasks.rados.rados.0.smithi121.stdout:307: finishing write tid 1 to smithi12138290-32 2024-09-16T03:41:01.450 INFO:tasks.rados.rados.0.smithi121.stdout:307: finishing write tid 2 to smithi12138290-32 2024-09-16T03:41:01.450 INFO:tasks.rados.rados.0.smithi121.stdout:307: finishing write tid 3 to smithi12138290-32 2024-09-16T03:41:01.450 INFO:tasks.rados.rados.0.smithi121.stdout:311: finishing rollback tid 0 to smithi12138290-4 2024-09-16T03:41:01.450 INFO:tasks.rados.rados.0.smithi121.stdout:308: done (6 left) 2024-09-16T03:41:01.450 INFO:tasks.rados.rados.0.smithi121.stdout:310: done (5 left) 2024-09-16T03:41:01.450 INFO:tasks.rados.rados.0.smithi121.stdout:314: snap_remove snap 14 2024-09-16T03:41:01.480 INFO:tasks.rados.rados.0.smithi121.stdout:314: done (5 left) 2024-09-16T03:41:01.480 INFO:tasks.rados.rados.0.smithi121.stdout:315: snap_create 2024-09-16T03:41:01.480 INFO:tasks.rados.rados.0.smithi121.stdout:311: finishing rollback tid 1 to smithi12138290-4 2024-09-16T03:41:01.480 INFO:tasks.rados.rados.0.smithi121.stdout:311: finishing rollback tid 2 to smithi12138290-4 2024-09-16T03:41:01.481 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 4 v 63 (ObjNum 79 snap 11 seq_num 79) dirty exists 2024-09-16T03:41:01.481 INFO:tasks.rados.rados.0.smithi121.stdout:307: finishing write tid 4 to smithi12138290-32 2024-09-16T03:41:01.481 INFO:tasks.rados.rados.0.smithi121.stdout:307: finishing write tid 5 to smithi12138290-32 2024-09-16T03:41:01.481 INFO:tasks.rados.rados.0.smithi121.stdout:309: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-16T03:41:01.539 INFO:tasks.rados.rados.0.smithi121.stdout:307: finishing write tid 6 to smithi12138290-32 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:307: oid 32 updating version 0 to 138 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:307: oid 32 version 138 is already newer than 136 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:307: oid 32 version 138 is already newer than 137 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:307: oid 32 updating version 138 to 139 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:307: oid 32 updating version 139 to 140 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 32 v 140 (ObjNum 139 snap 27 seq_num 139) dirty exists 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:307: left oid 32 (ObjNum 139 snap 27 seq_num 139) 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:312: finishing write tid 1 to smithi12138290-19 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:312: finishing write tid 2 to smithi12138290-19 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:312: finishing write tid 3 to smithi12138290-19 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:313: finishing rollback tid 0 to smithi12138290-16 2024-09-16T03:41:01.540 INFO:tasks.rados.rados.0.smithi121.stdout:312: finishing write tid 4 to smithi12138290-19 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:313: finishing rollback tid 1 to smithi12138290-16 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:313: finishing rollback tid 2 to smithi12138290-16 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 16 v 72 (ObjNum 96 snap 15 seq_num 96) dirty exists 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:312: finishing write tid 5 to smithi12138290-19 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:312: finishing write tid 6 to smithi12138290-19 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:312: oid 19 updating version 0 to 128 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:312: oid 19 updating version 128 to 129 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:312: oid 19 version 129 is already newer than 127 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:312: oid 19 version 129 is already newer than 125 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:312: oid 19 version 129 is already newer than 126 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 19 v 129 (ObjNum 140 snap 27 seq_num 140) dirty exists 2024-09-16T03:41:01.541 INFO:tasks.rados.rados.0.smithi121.stdout:312: left oid 19 (ObjNum 140 snap 27 seq_num 140) 2024-09-16T03:41:01.542 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:01.537+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:01.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:41:01.607 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:01.607 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:01.617 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:41:01.624 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:41:01.624 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:41:01.640 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:01.638+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:01.650 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:41:01.650 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:41:01.668 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:41:01.668 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:41:01.685 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:41:01.685 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:41:01.708 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:41:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:41:01.726 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:41:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:41:01.733 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 54/31 ECSubWrite(tid=57, reqid=osd.6.0:56, at_version=0'0, trim_to=0'0, roll_forward_to=54'45))", 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:33.518441+0000", 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.212654194999999, 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006708416, 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518441+0000", 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518441+0000", 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518442+0000", 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2200000000000003e-07 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518454+0000", 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2285e-05 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518465+0000", 2024-09-16T03:41:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0569e-05 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518470+0000", 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7150000000000003e-06 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518512+0000", 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1548000000000003e-05 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518518+0000", 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.331e-06 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525112+0000", 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065939220000000003 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525149+0000", 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7323999999999997e-05 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.0s2 54/31 ECSubWrite(tid=30, reqid=client.4615.0:421, at_version=0'0, trim_to=0'0, roll_forward_to=54'16))", 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:33.518762+0000", 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.212332855, 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063986420000000004, 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518762+0000", 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518762+0000", 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518763+0000", 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.13e-07 2024-09-16T03:41:01.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518773+0000", 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0448999999999999e-05 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518783+0000", 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5279999999999992e-06 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518788+0000", 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3430000000000003e-06 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518830+0000", 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1984000000000003e-05 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.518836+0000", 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.105e-06 2024-09-16T03:41:01.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525124+0000", 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062873820000000002 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:33.525161+0000", 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7237999999999998e-05 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 73/55 ECSubWrite(tid=24, reqid=osd.3.0:24, at_version=73'39, trim_to=0'0, roll_forward_to=72'35))", 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.478776+0000", 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2523193399999997, 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085502010000000003, 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478776+0000", 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478776+0000", 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478776+0000", 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7199999999999999e-07 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478796+0000", 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9762000000000002e-05 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478807+0000", 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0368e-05 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478814+0000", 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5379999999999997e-06 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478866+0000", 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1960999999999999e-05 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478875+0000", 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3600000000000002e-06 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487290+0000", 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008414932 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487326+0000", 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5707999999999998e-05 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.0s2 73/31 ECSubWrite(tid=85, reqid=osd.1.0:85, at_version=73'40, trim_to=0'0, roll_forward_to=72'36))", 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.479171+0000", 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2519240089999997, 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080606229999999994, 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479171+0000", 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479171+0000", 2024-09-16T03:41:01.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479172+0000", 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7899999999999998e-07 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479191+0000", 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8802999999999999e-05 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479202+0000", 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0865e-05 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479209+0000", 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6809999999999998e-06 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479259+0000", 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.922e-05 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479269+0000", 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0017000000000001e-05 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487209+0000", 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079406809999999998 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487232+0000", 2024-09-16T03:41:01.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2677e-05 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 73/31 ECSubWrite(tid=110, reqid=osd.6.0:110, at_version=73'89, trim_to=0'0, roll_forward_to=72'85))", 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.480970+0000", 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2501256420000004, 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089551219999999994, 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.753 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480970+0000", 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480970+0000", 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480971+0000", 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8400000000000003e-07 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480991+0000", 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0577999999999998e-05 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481005+0000", 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4369000000000001e-05 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481014+0000", 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481088+0000", 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4051e-05 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481098+0000", 2024-09-16T03:41:01.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7820000000000005e-06 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489895+0000", 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087973840000000001 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489925+0000", 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9388000000000001e-05 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.757 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.as1 73/55 ECSubWrite(tid=117, reqid=osd.12.0:114, at_version=0'0, trim_to=0'0, roll_forward_to=73'28))", 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.480999+0000", 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2500959609999995, 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061571000000000004, 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480999+0000", 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480999+0000", 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481000+0000", 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481011+0000", 2024-09-16T03:41:01.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1134999999999999e-05 2024-09-16T03:41:01.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481023+0000", 2024-09-16T03:41:01.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1773999999999999e-05 2024-09-16T03:41:01.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481036+0000", 2024-09-16T03:41:01.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3115000000000001e-05 2024-09-16T03:41:01.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481103+0000", 2024-09-16T03:41:01.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6927999999999993e-05 2024-09-16T03:41:01.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481114+0000", 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0786e-05 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487123+0000", 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060096079999999996 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487156+0000", 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2997999999999998e-05 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds1 73/31 ECSubWrite(tid=119, reqid=osd.12.0:119, at_version=73'56, trim_to=0'0, roll_forward_to=73'54))", 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.481659+0000", 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2494366840000009, 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082272739999999997, 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481659+0000", 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481659+0000", 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481659+0000", 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1699999999999998e-07 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481677+0000", 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7518999999999999e-05 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481686+0000", 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3340000000000008e-06 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481694+0000", 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6450000000000005e-06 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481750+0000", 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6385000000000003e-05 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481760+0000", 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.842e-06 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489853+0000", 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080927240000000008 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489886+0000", 2024-09-16T03:41:01.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3308e-05 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.7s1 73/55 ECSubWrite(tid=55, reqid=osd.9.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=73'41))", 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.483812+0000", 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2472829149999995, 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065191329999999999, 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.768 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483812+0000", 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483812+0000", 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483813+0000", 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3200000000000004e-07 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483822+0000", 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1770000000000004e-06 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483832+0000", 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8819999999999996e-06 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483840+0000", 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5120000000000002e-06 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483897+0000", 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7479000000000003e-05 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483906+0000", 2024-09-16T03:41:01.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1139999999999995e-06 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490296+0000", 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063896789999999997 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490331+0000", 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5558000000000001e-05 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.772 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 73/31 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=73'31, trim_to=0'0, roll_forward_to=66'29))", 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.484484+0000", 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2466115440000003, 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085472250000000003, 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484484+0000", 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484484+0000", 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484484+0000", 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8200000000000001e-07 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484507+0000", 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2535999999999999e-05 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484517+0000", 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.084e-05 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484537+0000", 2024-09-16T03:41:01.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9174e-05 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484602+0000", 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5584000000000001e-05 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484611+0000", 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1339999999999993e-06 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.493014+0000", 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084031120000000008 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.493031+0000", 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6463000000000001e-05 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.777 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 73/31 ECSubWrite(tid=93, reqid=osd.10.0:93, at_version=73'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.485354+0000", 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2457409360000007, 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076572819999999996, 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.485354+0000", 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.485354+0000", 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.485355+0000", 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4900000000000002e-07 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.485374+0000", 2024-09-16T03:41:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9320000000000001e-05 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.485387+0000", 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2716e-05 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.485396+0000", 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0680000000000003e-06 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.485442+0000", 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6554000000000002e-05 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.485449+0000", 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.849e-06 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.492995+0000", 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007546082 2024-09-16T03:41:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.493012+0000", 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6444e-05 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 73/31 ECSubWrite(tid=94, reqid=osd.10.0:94, at_version=73'42, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.487044+0000", 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2440517369999995, 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084560920000000001, 2024-09-16T03:41:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487044+0000", 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487044+0000", 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487044+0000", 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8899999999999999e-07 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487072+0000", 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7889999999999999e-05 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487082+0000", 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0098999999999999e-05 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487089+0000", 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2409999999999998e-06 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487733+0000", 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00064390699999999999 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487758+0000", 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4159999999999999e-05 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.495464+0000", 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077063560000000001 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.495500+0000", 2024-09-16T03:41:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5750000000000002e-05 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 73/31 ECSubWrite(tid=95, reqid=client.4615.0:649, at_version=73'44, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.488356+0000", 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2427391530000005, 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0074132579999999998, 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.488356+0000", 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.488356+0000", 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.488357+0000", 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8500000000000001e-07 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.488625+0000", 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000268261 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.488637+0000", 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1904e-05 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.488646+0000", 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0110000000000006e-06 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490153+0000", 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001507296 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490162+0000", 2024-09-16T03:41:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.495743+0000", 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055810369999999996 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.495769+0000", 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6412999999999999e-05 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.0s2 74/31 ECSubWrite(tid=101, reqid=osd.1.0:101, at_version=74'44, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.532923+0000", 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1981720970000005, 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079850240000000003, 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.532923+0000", 2024-09-16T03:41:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.532923+0000", 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.532924+0000", 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6100000000000001e-07 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.532945+0000", 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1725999999999998e-05 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.532954+0000", 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2239999999999995e-06 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.532960+0000", 2024-09-16T03:41:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7800000000000003e-06 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533019+0000", 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9126999999999998e-05 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533030+0000", 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0299000000000001e-05 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.540874+0000", 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078440090000000007 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.540908+0000", 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4397999999999998e-05 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.7s1 74/55 ECSubWrite(tid=62, reqid=osd.9.0:61, at_version=0'0, trim_to=0'0, roll_forward_to=74'44))", 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.533331+0000", 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1977642799999995, 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084000640000000005, 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533331+0000", 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533331+0000", 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533332+0000", 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.61e-07 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533342+0000", 2024-09-16T03:41:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0210000000000001e-05 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533352+0000", 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0195e-05 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533361+0000", 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1749999999999994e-06 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533419+0000", 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8029000000000003e-05 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533429+0000", 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.541696+0000", 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082673150000000008 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.541731+0000", 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4938000000000003e-05 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 74/55 ECSubWrite(tid=27, reqid=client.4615.0:683, at_version=74'42, trim_to=0'0, roll_forward_to=73'39))", 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.533459+0000", 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1976359829999996, 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085407710000000008, 2024-09-16T03:41:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533459+0000", 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533459+0000", 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533460+0000", 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533481+0000", 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0571999999999999e-05 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533493+0000", 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2116999999999999e-05 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533500+0000", 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6290000000000001e-06 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533588+0000", 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7546000000000005e-05 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533600+0000", 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1812e-05 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.541974+0000", 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083744969999999998 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542000+0000", 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5837999999999999e-05 2024-09-16T03:41:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds1 74/31 ECSubWrite(tid=126, reqid=client.4615.0:679, at_version=74'62, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.533660+0000", 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1974348070000005, 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083896070000000003, 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533660+0000", 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533660+0000", 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533661+0000", 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:41:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533680+0000", 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8777e-05 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533691+0000", 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1354e-05 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533699+0000", 2024-09-16T03:41:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7870000000000002e-06 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533777+0000", 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7498999999999998e-05 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533787+0000", 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542029+0000", 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082426200000000008 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542050+0000", 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0886e-05 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 74/31 ECSubWrite(tid=106, reqid=osd.10.0:106, at_version=74'52, trim_to=0'0, roll_forward_to=73'50))", 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.534506+0000", 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1965889759999992, 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090150509999999996, 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534506+0000", 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534506+0000", 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534507+0000", 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.91e-07 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534542+0000", 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5018999999999998e-05 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534551+0000", 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6889999999999999e-06 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534560+0000", 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3040000000000005e-06 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534624+0000", 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4298e-05 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534633+0000", 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5759999999999995e-06 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.543482+0000", 2024-09-16T03:41:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088489710000000006 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.543521+0000", 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8902999999999997e-05 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.as1 74/55 ECSubWrite(tid=127, reqid=osd.12.0:125, at_version=0'0, trim_to=0'0, roll_forward_to=74'30))", 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.535677+0000", 2024-09-16T03:41:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1954178500000001, 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062379690000000003, 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.535677+0000", 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.535677+0000", 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.535678+0000", 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0999999999999997e-07 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.535692+0000", 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3562e-05 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.535703+0000", 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1491999999999999e-05 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.535712+0000", 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3559999999999993e-06 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536114+0000", 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040276199999999998 2024-09-16T03:41:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536126+0000", 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1354e-05 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.541872+0000", 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057463790000000002 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.541915+0000", 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3254000000000003e-05 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 74/55 ECSubWrite(tid=29, reqid=osd.3.0:29, at_version=74'45, trim_to=0'0, roll_forward_to=73'39))", 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.536364+0000", 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1947314999999996, 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071416409999999998, 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536364+0000", 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536364+0000", 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536364+0000", 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536384+0000", 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0098e-05 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536393+0000", 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4290000000000001e-06 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536399+0000", 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.849e-06 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536737+0000", 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033756699999999999 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536746+0000", 2024-09-16T03:41:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.772e-06 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.543471+0000", 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067249390000000001 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.543505+0000", 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4654999999999999e-05 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 74/55 ECSubWrite(tid=30, reqid=osd.3.0:30, at_version=74'47, trim_to=0'0, roll_forward_to=73'39))", 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.536410+0000", 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1946856619999995, 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089080830000000007, 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536410+0000", 2024-09-16T03:41:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536410+0000", 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536410+0000", 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536430+0000", 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9772999999999999e-05 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536448+0000", 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8434e-05 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536455+0000", 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1069999999999999e-06 2024-09-16T03:41:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.540745+0000", 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0042902449999999998 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.540755+0000", 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3449999999999995e-06 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.545279+0000", 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045243419999999998 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.545318+0000", 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8510999999999998e-05 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:01.829 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:01.831 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:41:01.831 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:41:01.832 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:41:01.832 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:41:01.832 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:41:01.832 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:41:01.833 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:41:01.833 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:41:01.833 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:41:01.834 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:41:01.834 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:41:01.834 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:41:01.834 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:41:01.835 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:01.829+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:01.849 INFO:teuthology.orchestra.run.smithi076.stdout:osd.12: {} 2024-09-16T03:41:01.849 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:41:01.866 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:41:01.866 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:41:01.883 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:41:01.883 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:41:01.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:41:01.883+0000 7f82cd802640 0 -- 172.21.15.76:0/2267915668 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f82c8221050 msgr2=0x7f82c8223440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:41:01.901 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:41:01.901 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:41:01.931 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:01.929+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:01.940 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:01.940 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:01.940 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:41:01.940 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:01.949 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:41:02.032 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:41:02.030+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:41:02.092 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:41:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:41:02.092 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:02.099 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:41:02.142 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:41:02.141+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:41:02.204 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:02.204 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:41:02.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:41:02.204 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:41:02.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:229 3.5s0 3:a4bec39a:::smithi12138290-4:head [copy-from ver 9] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 44.661141344999997, 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016562443, 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 229 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540579+0000", 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6910000000000002e-06 2024-09-16T03:41:02.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540591+0000", 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2018000000000001e-05 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540631+0000", 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9938999999999998e-05 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540643+0000", 2024-09-16T03:41:02.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1902000000000001e-05 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540731+0000", 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8585000000000007e-05 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540736+0000", 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9919999999999998e-06 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542533+0000", 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017970950000000001 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542727+0000", 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019304700000000001 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.553519+0000", 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010792568000000001 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554988+0000", 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014687490000000001 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557059+0000", 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00207092 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557139+0000", 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9937000000000004e-05 2024-09-16T03:41:02.210 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:230 3.5s0 3:a4bec39a:::smithi12138290-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 44.661030240000002, 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014084419000000001, 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 230 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540688+0000", 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540692+0000", 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.019e-06 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540724+0000", 2024-09-16T03:41:02.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2237000000000003e-05 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540740+0000", 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6075000000000001e-05 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540783+0000", 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2249000000000002e-05 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540786+0000", 2024-09-16T03:41:02.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4869999999999998e-06 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543015+0000", 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022290859999999999 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543063+0000", 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7496000000000001e-05 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554628+0000", 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011565203 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554697+0000", 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9381999999999994e-05 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554772+0000", 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4535000000000004e-05 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.216 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:314 3.5s0 3:a4bec39a:::smithi12138290-4:head [stat,delete] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.710964248000003, 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010497464, 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 314 2024-09-16T03:41:02.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490755+0000", 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.736e-06 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490767+0000", 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1931e-05 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490783+0000", 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6135000000000001e-05 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490791+0000", 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1620000000000008e-06 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490849+0000", 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7581000000000003e-05 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490856+0000", 2024-09-16T03:41:02.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9299999999999997e-06 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.492657+0000", 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018011539999999999 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.492731+0000", 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4263000000000004e-05 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.493469+0000", 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073743400000000003 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.495358+0000", 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001888898 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.501195+0000", 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058370879999999998 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.501251+0000", 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6152e-05 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=27, reqid=client.4615.0:354, at_version=53'8, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 29.696766718999999, 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055158589999999997, 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.223 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1e-07 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504976+0000", 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5259999999999999e-05 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504984+0000", 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6850000000000001e-06 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504993+0000", 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4299999999999995e-06 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505310+0000", 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031660299999999999 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505342+0000", 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.15e-05 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510423+0000", 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050819000000000003 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510467+0000", 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3271e-05 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.227 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:467 3.5s0 3:a45e0081:::smithi12138290-25:head [copy-from ver 16] snapc 11=[11,10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.718886504, 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014721097000000001, 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 467 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482832+0000", 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3e-06 2024-09-16T03:41:02.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482839+0000", 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3010000000000003e-06 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482849+0000", 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9499999999999996e-06 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482854+0000", 2024-09-16T03:41:02.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1410000000000004e-06 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482916+0000", 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2040999999999998e-05 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483028+0000", 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112674 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.231 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.491088+0000", 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080595969999999999 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.493079+0000", 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019907990000000001 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.497458+0000", 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004378804 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.497552+0000", 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4489999999999998e-05 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:468 3.5s0 3:a45e0081:::smithi12138290-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.718834524999998, 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010159837999999999, 2024-09-16T03:41:02.233 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 468 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482884+0000", 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.16e-07 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482888+0000", 2024-09-16T03:41:02.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0099999999999997e-06 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482903+0000", 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5418999999999999e-05 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482910+0000", 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6850000000000004e-06 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483315+0000", 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040511699999999998 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483369+0000", 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3884000000000001e-05 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492657+0000", 2024-09-16T03:41:02.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092879590000000001 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492897+0000", 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023993300000000001 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.493043+0000", 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014601499999999999 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.fs1 73/55 ECSubWrite(tid=174, reqid=osd.0.0:174, at_version=73'30, trim_to=0'0, roll_forward_to=56'28))", 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.478077+0000", 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.7236401679999993, 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060656310000000001, 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478077+0000", 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478077+0000", 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478078+0000", 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478111+0000", 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3151000000000002e-05 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478122+0000", 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1420000000000001e-05 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478132+0000", 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7280000000000007e-06 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478185+0000", 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2837999999999998e-05 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478194+0000", 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6959999999999994e-06 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484096+0000", 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059024389999999998 2024-09-16T03:41:02.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484143+0000", 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7079000000000001e-05 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.as6 73/55 ECSubWrite(tid=117, reqid=osd.12.0:114, at_version=0'0, trim_to=0'0, roll_forward_to=73'28))", 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.480476+0000", 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.7212419069999996, 2024-09-16T03:41:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063949760000000001, 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480476+0000", 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480476+0000", 2024-09-16T03:41:02.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480476+0000", 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480486+0000", 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6309999999999998e-06 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480500+0000", 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3767999999999999e-05 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480508+0000", 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2530000000000004e-06 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480592+0000", 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3918999999999994e-05 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480601+0000", 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5130000000000002e-06 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.486838+0000", 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062362119999999997 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.486871+0000", 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2981999999999997e-05 2024-09-16T03:41:02.247 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 73/31 ECSubWrite(tid=118, reqid=osd.12.0:116, at_version=0'0, trim_to=0'0, roll_forward_to=73'54))", 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.480541+0000", 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.7211769570000008, 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064373980000000004, 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480541+0000", 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480541+0000", 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480541+0000", 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9900000000000002e-07 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480554+0000", 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3071000000000001e-05 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480563+0000", 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9889999999999998e-06 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480612+0000", 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8878000000000002e-05 2024-09-16T03:41:02.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480767+0000", 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015537400000000001 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480776+0000", 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9560000000000003e-06 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.486953+0000", 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061768220000000002 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.486978+0000", 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5009000000000001e-05 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 73/31 ECSubWrite(tid=119, reqid=osd.12.0:119, at_version=73'56, trim_to=0'0, roll_forward_to=73'54))", 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.481500+0000", 2024-09-16T03:41:02.252 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.7202171859999993, 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079937859999999993, 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481500+0000", 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481500+0000", 2024-09-16T03:41:02.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481501+0000", 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3499999999999995e-07 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481520+0000", 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8533999999999999e-05 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481540+0000", 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0389000000000001e-05 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481548+0000", 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1259999999999998e-06 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481613+0000", 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4640000000000005e-05 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481621+0000", 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5690000000000005e-06 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489462+0000", 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078409450000000002 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489494+0000", 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1847999999999997e-05 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.256 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 73/31 ECSubWrite(tid=91, reqid=osd.14.0:91, at_version=73'73, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.482669+0000", 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.7190486790000001, 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0068390079999999997, 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482669+0000", 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482669+0000", 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482669+0000", 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.54e-07 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482686+0000", 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.713e-05 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482694+0000", 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3049999999999993e-06 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482700+0000", 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0629999999999996e-06 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482759+0000", 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9052999999999999e-05 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482769+0000", 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0759e-05 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489458+0000", 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066889030000000004 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489508+0000", 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9641000000000001e-05 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.483429+0000", 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.7182884630000004, 2024-09-16T03:41:02.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072320709999999996, 2024-09-16T03:41:02.262 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.262 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:02.262 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483429+0000", 2024-09-16T03:41:02.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483429+0000", 2024-09-16T03:41:02.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483430+0000", 2024-09-16T03:41:02.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2799999999999996e-07 2024-09-16T03:41:02.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483443+0000", 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3332e-05 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483451+0000", 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.977e-06 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483456+0000", 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4510000000000002e-06 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484487+0000", 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001031129 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484495+0000", 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5280000000000003e-06 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490628+0000", 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061330730000000002 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490661+0000", 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2953000000000001e-05 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s1 73/31 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=73'31, trim_to=0'0, roll_forward_to=66'29))", 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.484560+0000", 2024-09-16T03:41:02.266 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.7171574659999997, 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057403109999999997, 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484560+0000", 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484560+0000", 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484560+0000", 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4900000000000002e-07 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484581+0000", 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0741999999999999e-05 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484592+0000", 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1087000000000001e-05 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484601+0000", 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6729999999999998e-06 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484654+0000", 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3606999999999997e-05 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484660+0000", 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7559999999999996e-06 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490261+0000", 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056009620000000001 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490300+0000", 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9235e-05 2024-09-16T03:41:02.270 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:685 3.5s0 3:a45e0081:::smithi12138290-25:head [read 0~1409024,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.529168+0000", 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.6725499910000003, 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070249730000000003, 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 685 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529168+0000", 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529168+0000", 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529170+0000", 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8700000000000001e-06 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529180+0000", 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3500000000000003e-06 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529203+0000", 2024-09-16T03:41:02.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3410000000000001e-05 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529214+0000", 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0360000000000001e-05 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529367+0000", 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015317600000000001 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529468+0000", 2024-09-16T03:41:02.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000101619 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536193+0000", 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067241879999999999 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:686 3.5s0 3:a45e0081:::smithi12138290-25:head [read 0~1409024,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.529236+0000", 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.6724811240000008, 2024-09-16T03:41:02.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090419060000000006, 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 686 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529236+0000", 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529236+0000", 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529237+0000", 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529241+0000", 2024-09-16T03:41:02.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2429999999999999e-06 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529249+0000", 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6229999999999997e-06 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529255+0000", 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6439999999999997e-06 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533107+0000", 2024-09-16T03:41:02.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038518879999999999 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533199+0000", 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2863999999999997e-05 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.538278+0000", 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050790059999999996 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:687 3.5s0 3:a45e0081:::smithi12138290-25:head [read 0~1409024,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.529263+0000", 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.6724545170000003, 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01182183, 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 687 2024-09-16T03:41:02.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529263+0000", 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529263+0000", 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529264+0000", 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3600000000000004e-07 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529267+0000", 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1190000000000001e-06 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529273+0000", 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4649999999999999e-06 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529278+0000", 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6059999999999996e-06 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536430+0000", 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071525820000000002 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536515+0000", 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4332999999999994e-05 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.541085+0000", 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045701889999999997 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 74/31 ECSubWrite(tid=128, reqid=client.4615.0:680, at_version=74'63, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.537219+0000", 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.664498944, 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056221450000000003, 2024-09-16T03:41:02.284 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537219+0000", 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537219+0000", 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537219+0000", 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-07 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537235+0000", 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5577e-05 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537243+0000", 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8049999999999996e-06 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537251+0000", 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.126e-06 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.538933+0000", 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001682006 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.538941+0000", 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8859999999999993e-06 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542806+0000", 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038642699999999999 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542841+0000", 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5105999999999999e-05 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.288 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:778 3.5s0 3:a4bec39a:::smithi12138290-4:1b [read 0~1343488,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:56.583639+0000", 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.6180783779999999, 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007324077, 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 778 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583639+0000", 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583639+0000", 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583641+0000", 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8619999999999999e-06 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583648+0000", 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8859999999999998e-06 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583665+0000", 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6962000000000001e-05 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583675+0000", 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0127e-05 2024-09-16T03:41:02.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583740+0000", 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5353999999999998e-05 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583845+0000", 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104431 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.590963+0000", 2024-09-16T03:41:02.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071184550000000001 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:779 3.5s0 3:a4bec39a:::smithi12138290-4:1b [read 0~1343488,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:56.583697+0000", 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.6180206269999999, 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079483090000000006, 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.293 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.294 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 779 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583697+0000", 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583697+0000", 2024-09-16T03:41:02.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583698+0000", 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4499999999999997e-07 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583701+0000", 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7790000000000002e-06 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583709+0000", 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2120000000000003e-06 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583715+0000", 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7579999999999998e-06 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.584246+0000", 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00053043999999999997 2024-09-16T03:41:02.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.584344+0000", 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8442000000000004e-05 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.591645+0000", 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073011329999999996 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:780 3.5s0 3:a4bec39a:::smithi12138290-4:1b [read 0~1343488,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:56.583722+0000", 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.6179957710000004, 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085727600000000004, 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 780 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583722+0000", 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583722+0000", 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583722+0000", 2024-09-16T03:41:02.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-07 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583725+0000", 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9699999999999999e-06 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583743+0000", 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7632e-05 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583748+0000", 2024-09-16T03:41:02.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7570000000000004e-06 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.586247+0000", 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024994029999999999 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.586333+0000", 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6497999999999998e-05 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.592295+0000", 2024-09-16T03:41:02.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005961142 2024-09-16T03:41:02.304 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.304 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.304 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.304 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.304 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.304 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:02.306 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:41:02.306 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:41:02.304+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:41:02.406 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:02.404+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:02.451 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:02.451 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:02.451 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:41:02.451 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:02.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:41:02.475 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T03:41:02.481 INFO:tasks.rados.rados.0.smithi121.stdout:307: done (5 left) 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:309: done (4 left) 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:311: done (3 left) 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:312: done (2 left) 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:313: done (1 left) 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:315: done (0 left) 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:316: rollback oid 33 current snap is 28 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 33 to 25 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:317: read oid 38 snap -1 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:317: expect deleted 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:318: rollback oid 9 current snap is 28 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 9 to 9 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:319: rollback oid 50 current snap is 28 2024-09-16T03:41:02.482 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 50 to 6 2024-09-16T03:41:02.483 INFO:tasks.rados.rados.0.smithi121.stdout:320: delete oid 23 current snap is 28 2024-09-16T03:41:02.488 INFO:tasks.rados.rados.0.smithi121.stdout:317: done (4 left) 2024-09-16T03:41:02.488 INFO:tasks.rados.rados.0.smithi121.stdout:320: done (3 left) 2024-09-16T03:41:02.488 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 14 current snap is 28 2024-09-16T03:41:02.488 INFO:tasks.rados.rados.0.smithi121.stdout:321: seq_num 142 ranges {3129344=442368,3571712=720896,4292608=540672,4833280=688128} 2024-09-16T03:41:02.507 INFO:tasks.rados.rados.0.smithi121.stdout:321: writing smithi12138290-14 from 3129344 to 3571712 tid 1 2024-09-16T03:41:02.511 INFO:tasks.rados.rados.0.smithi121.stdout:321: writing smithi12138290-14 from 3571712 to 4292608 tid 2 2024-09-16T03:41:02.514 INFO:tasks.rados.rados.0.smithi121.stdout:321: writing smithi12138290-14 from 4292608 to 4833280 tid 3 2024-09-16T03:41:02.519 INFO:tasks.rados.rados.0.smithi121.stdout:321: writing smithi12138290-14 from 4833280 to 5521408 tid 4 2024-09-16T03:41:02.519 INFO:tasks.rados.rados.0.smithi121.stdout:322: snap_remove snap 23 2024-09-16T03:41:02.533 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:41:02.532+0000 7fa9bc3ee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33493) UID: 0 2024-09-16T03:41:02.568 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:02.569 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:02.569 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:41:02.569 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:41:02.569 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:02.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:41:02.609 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:02.607+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:02.709 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:02.708+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:02.724 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:02.724 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:41:02.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:41:02.724 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:41:02.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.724 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:253 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:41:02.724 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.142146171, 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084206999999999997, 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 253 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579058+0000", 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579061+0000", 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3350000000000001e-06 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579072+0000", 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0964999999999999e-05 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579093+0000", 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1766000000000001e-05 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579104+0000", 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0271999999999999e-05 2024-09-16T03:41:02.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579200+0000", 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5917999999999999e-05 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579208+0000", 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9479999999999997e-06 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581270+0000", 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020611140000000002 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581437+0000", 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016734000000000001 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587479+0000", 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060420420000000001 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.729 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:254 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.142063084999997, 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087869469999999998, 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.730 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 254 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579141+0000", 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579142+0000", 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579146+0000", 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0600000000000001e-06 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579155+0000", 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7620000000000001e-06 2024-09-16T03:41:02.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579161+0000", 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8640000000000001e-06 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579289+0000", 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000128677 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579298+0000", 2024-09-16T03:41:02.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4100000000000008e-06 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581916+0000", 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026177570000000001 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.581995+0000", 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9833999999999995e-05 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.587928+0000", 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059329329999999996 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:255 3.ds0 3:b952e66c:::smithi12138290-29:head [read 0~475136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.142019167999997, 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090005769999999992, 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.735 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 255 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579185+0000", 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579186+0000", 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9500000000000002e-07 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579189+0000", 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579197+0000", 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8189999999999992e-06 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579202+0000", 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579337+0000", 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134547 2024-09-16T03:41:02.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.579343+0000", 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583029+0000", 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036861540000000001 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.583113+0000", 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3370999999999998e-05 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:21.588186+0000", 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050730269999999999 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:354 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:02.740 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.498416+0000", 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.222788690000002, 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017750095, 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 354 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498416+0000", 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498416+0000", 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498418+0000", 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0899999999999999e-06 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498426+0000", 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.909e-06 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498442+0000", 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6164999999999999e-05 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498451+0000", 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5240000000000006e-06 2024-09-16T03:41:02.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498523+0000", 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1482000000000004e-05 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498535+0000", 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2072000000000001e-05 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502229+0000", 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003693806 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.502333+0000", 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104041 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503454+0000", 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011210440000000001 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515617+0000", 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012162681999999999 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516067+0000", 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000450817 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516166+0000", 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8462999999999996e-05 2024-09-16T03:41:02.746 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:355 3.ds0 3:b952e66c:::smithi12138290-29:head [rollback b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.222729818000001, 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017895208999999999, 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.747 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.748 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 355 2024-09-16T03:41:02.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.748 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:41:02.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498475+0000", 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8100000000000003e-07 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498478+0000", 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8049999999999998e-06 2024-09-16T03:41:02.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498485+0000", 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0219999999999999e-06 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498489+0000", 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9829999999999998e-06 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498604+0000", 2024-09-16T03:41:02.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011544699999999999 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498609+0000", 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2760000000000002e-06 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.503989+0000", 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053800360000000004 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504091+0000", 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010255300000000001 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504699+0000", 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00060748699999999998 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515811+0000", 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011112662000000001 2024-09-16T03:41:02.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516302+0000", 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049017500000000003 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516370+0000", 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8281999999999994e-05 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:356 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.498505+0000", 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.222699168999998, 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018089424, 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 356 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498505+0000", 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498505+0000", 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498506+0000", 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3700000000000001e-07 2024-09-16T03:41:02.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498508+0000", 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498513+0000", 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4519999999999996e-06 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498518+0000", 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6399999999999996e-06 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498643+0000", 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125265 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.498648+0000", 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0320000000000003e-06 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505217+0000", 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065689839999999999 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505317+0000", 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9613999999999999e-05 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505895+0000", 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057821100000000002 2024-09-16T03:41:02.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.515868+0000", 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099727659999999992 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516509+0000", 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00064155600000000005 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516595+0000", 2024-09-16T03:41:02.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5206999999999999e-05 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:357 3.ds0 3:b84aed6c:::smithi12138290-13:head [append 0~622592] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.504582+0000", 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.216622597000001, 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012468976999999999, 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.760 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 357 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504582+0000", 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504582+0000", 2024-09-16T03:41:02.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504584+0000", 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.057e-06 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505874+0000", 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012901060000000001 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505888+0000", 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3689999999999999e-05 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505896+0000", 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3580000000000003e-06 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506261+0000", 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000365239 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.506357+0000", 2024-09-16T03:41:02.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5466999999999999e-05 2024-09-16T03:41:02.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.508262+0000", 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001905622 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516183+0000", 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079204419999999998 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516969+0000", 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00078583600000000004 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517051+0000", 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2159999999999999e-05 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:358 3.ds0 3:b84aed6c:::smithi12138290-13:head [append 0~475136] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.507540+0000", 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.213664810000001, 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011581542, 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.766 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 358 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507540+0000", 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507540+0000", 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507541+0000", 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.716e-06 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507949+0000", 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040747899999999998 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507963+0000", 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3828999999999999e-05 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.507971+0000", 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7430000000000008e-06 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510078+0000", 2024-09-16T03:41:02.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002106293 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510200+0000", 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012225100000000001 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.511236+0000", 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010360289999999999 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516185+0000", 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049487079999999996 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519053+0000", 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028678610000000002 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519121+0000", 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8633000000000005e-05 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.771 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:359 3.ds0 3:b84aed6c:::smithi12138290-13:head [append 0~294912] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.509631+0000", 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.211572997000001, 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097547659999999998, 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 359 2024-09-16T03:41:02.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509631+0000", 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509631+0000", 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509634+0000", 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1459999999999999e-06 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509870+0000", 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023675700000000001 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509882+0000", 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1905999999999999e-05 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509891+0000", 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9870000000000005e-06 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512302+0000", 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002410812 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.512443+0000", 2024-09-16T03:41:02.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014075700000000001 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.513286+0000", 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000843021 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.516670+0000", 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033843889999999998 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519344+0000", 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026737129999999999 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519386+0000", 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2277999999999997e-05 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:360 3.ds0 3:b84aed6c:::smithi12138290-13:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:41:02.777 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.21129578, 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010089805, 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 360 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:41:02.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509909+0000", 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8999999999999996e-07 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509913+0000", 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0269999999999997e-06 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509922+0000", 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.687e-06 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509928+0000", 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8000000000000004e-06 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517293+0000", 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073646559999999998 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517402+0000", 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109687 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.517900+0000", 2024-09-16T03:41:02.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049748700000000002 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519217+0000", 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001317033 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519941+0000", 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00072411000000000003 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.519998+0000", 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7627999999999998e-05 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:361 3.ds0 3:b84aed6c:::smithi12138290-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.509950+0000", 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.211253963000001, 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01277462, 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 361 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509950+0000", 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509950+0000", 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509951+0000", 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7299999999999996e-07 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509956+0000", 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4739999999999999e-06 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509963+0000", 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.509968+0000", 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4530000000000001e-06 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518702+0000", 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087336250000000001 2024-09-16T03:41:02.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.518806+0000", 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104905 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.520329+0000", 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015226879999999999 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.520422+0000", 2024-09-16T03:41:02.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3224000000000001e-05 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.522725+0000", 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023026980000000002 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:578 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~655360] snapc 17=[17,16,15,13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:47.482693+0000", 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.238511669999999, 2024-09-16T03:41:02.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093753210000000007, 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 578 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.482693+0000", 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.482693+0000", 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.482695+0000", 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5000000000000002e-06 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.483400+0000", 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00070434899999999995 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.483422+0000", 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1963999999999998e-05 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.483434+0000", 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2867000000000001e-05 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.483544+0000", 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109376 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.483621+0000", 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6940000000000005e-05 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.484484+0000", 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00086355600000000002 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.488177+0000", 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00369251 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492014+0000", 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038367269999999998 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:47.492068+0000", 2024-09-16T03:41:02.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4531999999999999e-05 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:601 3.ds0 3:b433db86:::smithi12138290-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:48.525144+0000", 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.196060319000001, 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073124959999999999, 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.794 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 601 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.525144+0000", 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.525144+0000", 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.525144+0000", 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.525147+0000", 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.351e-06 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.525150+0000", 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.525153+0000", 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.527106+0000", 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001952346 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.527188+0000", 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2818999999999994e-05 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.531000+0000", 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00381183 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.531082+0000", 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2023999999999999e-05 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:48.532457+0000", 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013743099999999999 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:673 3.ds0 3:b952e66c:::smithi12138290-29:head [rollback 18] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.565818+0000", 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.155386761999999, 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010539477, 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 673 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565818+0000", 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565818+0000", 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565818+0000", 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0800000000000004e-07 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565821+0000", 2024-09-16T03:41:02.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0960000000000001e-06 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565830+0000", 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9479999999999997e-06 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565837+0000", 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1210000000000001e-06 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.567045+0000", 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012077629999999999 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.567139+0000", 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3776999999999993e-05 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.567687+0000", 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054761700000000003 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.570346+0000", 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026592709999999999 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.576306+0000", 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059596570000000001 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.576357+0000", 2024-09-16T03:41:02.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1619000000000002e-05 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:674 3.ds0 3:b952e66c:::smithi12138290-29:head [append 0~0] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.565856+0000", 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.155348912999999, 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010698042, 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 674 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565856+0000", 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565856+0000", 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565856+0000", 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.13e-07 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565859+0000", 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.999e-06 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565867+0000", 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8390000000000007e-06 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.565877+0000", 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0397999999999999e-05 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.568106+0000", 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022287370000000002 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.568195+0000", 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8745000000000005e-05 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.568626+0000", 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043139900000000001 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.570632+0000", 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002005246 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.576474+0000", 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058420579999999998 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.576554+0000", 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0007999999999997e-05 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:679 3.ds0 3:b433db86:::smithi12138290-41:head [append 0~0] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.527521+0000", 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1936836159999995, 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015492763, 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 679 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527521+0000", 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527521+0000", 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527523+0000", 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0930000000000001e-06 2024-09-16T03:41:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527539+0000", 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6035000000000001e-05 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527561+0000", 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2036999999999999e-05 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527571+0000", 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T03:41:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527642+0000", 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0555000000000005e-05 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527650+0000", 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1829999999999994e-06 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.531194+0000", 2024-09-16T03:41:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035441280000000001 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.531296+0000", 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000101677 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.532521+0000", 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001225599 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542083+0000", 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095617549999999999 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542921+0000", 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00083742699999999996 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.543014+0000", 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3023999999999996e-05 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:680 3.ds0 3:b433db86:::smithi12138290-41:head [rollback b] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.527598+0000", 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1936069029999992, 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017243158000000001, 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 680 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527598+0000", 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527598+0000", 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527598+0000", 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527604+0000", 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8499999999999999e-06 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527612+0000", 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2050000000000002e-06 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527618+0000", 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8540000000000002e-06 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527712+0000", 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3572999999999993e-05 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527718+0000", 2024-09-16T03:41:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.302e-06 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534867+0000", 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071490640000000001 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534974+0000", 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000106964 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.535718+0000", 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074355099999999996 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542141+0000", 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064238209999999997 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.544760+0000", 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026189030000000001 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.544841+0000", 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0407000000000005e-05 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:681 3.ds0 3:b433db86:::smithi12138290-41:head [append 0~0] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.527627+0000", 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1935775710000005, 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017531596, 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 681 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527627+0000", 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527627+0000", 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527627+0000", 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3000000000000001e-07 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527630+0000", 2024-09-16T03:41:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6850000000000001e-06 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527636+0000", 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2840000000000003e-06 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527641+0000", 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1379999999999998e-06 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527742+0000", 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100999 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.527748+0000", 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3199999999999999e-06 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536381+0000", 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086334700000000007 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536482+0000", 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100703 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536984+0000", 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00050255600000000003 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542142+0000", 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051579110000000003 2024-09-16T03:41:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.545092+0000", 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029494209999999998 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.545158+0000", 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6679000000000003e-05 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:806 3.as0 3:58a4fdf3:::smithi12138290-5:head [append 0~720896] snapc 1c=[1c,1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:58.561076+0000", 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.1601284249999999, 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072688300000000004, 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 806 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.561076+0000", 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.561076+0000", 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.561079+0000", 2024-09-16T03:41:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.841e-06 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.561906+0000", 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00082717000000000001 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.561929+0000", 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2721000000000002e-05 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.561938+0000", 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6229999999999992e-06 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.562019+0000", 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0871000000000005e-05 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.562103+0000", 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3529999999999995e-05 2024-09-16T03:41:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.563325+0000", 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001222648 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.567609+0000", 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0042832369999999996 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.568255+0000", 2024-09-16T03:41:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00064628199999999998 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.568345+0000", 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9907000000000005e-05 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:811 3.as0 3:58a4fdf3:::smithi12138290-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:58.570478+0000", 2024-09-16T03:41:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.150726551, 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010127371, 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 811 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:02.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.570478+0000", 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.570478+0000", 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.570478+0000", 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3000000000000001e-07 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.570480+0000", 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9800000000000001e-06 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.570485+0000", 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6809999999999997e-06 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.570489+0000", 2024-09-16T03:41:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.569e-06 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.576425+0000", 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059361509999999998 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.576538+0000", 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112902 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.578043+0000", 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015055489999999999 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.578129+0000", 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6014999999999996e-05 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.580605+0000", 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024760939999999999 2024-09-16T03:41:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.841 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.841 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.841 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:02.841 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:02.841 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:02.842 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:41:02.843 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:02.840+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:02.944 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:02.942+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:02.959 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:02.959 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:02.959 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:41:02.959 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:02.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:41:03.045 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:03.043+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:03.071 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:03.072 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:03.072 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:41:03.072 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:41:03.072 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:03.080 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:41:03.146 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:03.143+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:03.156 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15] out_osds: [13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 2, 6, 10, 14, 3, 7, 11, 15, 13] 2024-09-16T03:41:03.156 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:41:03.156 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-09-16T03:41:03.157 INFO:tasks.thrashosds.thrasher:Killing osd 12, live_osds are [0, 4, 8, 12, 1, 5, 9, 2, 6, 10, 14, 3, 7, 11, 15, 13] 2024-09-16T03:41:03.157 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-16T03:41:03.157 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:03.182 INFO:tasks.ceph.osd.12:Stopped 2024-09-16T03:41:03.182 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 12 2024-09-16T03:41:03.186 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:251 3.0s0 3:00cc7e13:::smithi12138290-42:head [copy-from ver 4] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.627218872999997, 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013120588000000001, 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.187 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 251 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:41:03.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556738+0000", 2024-09-16T03:41:03.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8989999999999999e-06 2024-09-16T03:41:03.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556745+0000", 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556759+0000", 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.451e-05 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556766+0000", 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1260000000000003e-06 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556817+0000", 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1693999999999997e-05 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556824+0000", 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8549999999999996e-06 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557940+0000", 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001115888 2024-09-16T03:41:03.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558009+0000", 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8707999999999993e-05 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564426+0000", 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064170659999999999 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.566682+0000", 2024-09-16T03:41:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022565039999999999 2024-09-16T03:41:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569779+0000", 2024-09-16T03:41:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030962770000000001 2024-09-16T03:41:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569856+0000", 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7843999999999995e-05 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:336 3.0s0 3:0cbe5f28:::smithi12138290-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:31.479494+0000", 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.704460461, 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01140001, 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.194 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 336 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479494+0000", 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479494+0000", 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479495+0000", 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4900000000000001e-07 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479497+0000", 2024-09-16T03:41:03.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1260000000000001e-06 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479503+0000", 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.479506+0000", 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.482263+0000", 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027574219999999998 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.482312+0000", 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9196999999999997e-05 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.489119+0000", 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068063610000000004 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.489197+0000", 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8570000000000002e-05 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:31.490894+0000", 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001697034 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:03.199 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.483445+0000", 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.700509228, 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0098327740000000007, 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.483445+0000", 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.483445+0000", 2024-09-16T03:41:03.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.483446+0000", 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.483462+0000", 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.483472+0000", 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7580000000000004e-06 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.483478+0000", 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9739999999999999e-06 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.483534+0000", 2024-09-16T03:41:03.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6353000000000001e-05 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.483540+0000", 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7230000000000001e-06 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493237+0000", 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096973249999999997 2024-09-16T03:41:03.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493278+0000", 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0955999999999999e-05 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.3s4 73/31 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=73'31, trim_to=0'0, roll_forward_to=66'29))", 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.484581+0000", 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.699373727999999, 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012587574000000001, 2024-09-16T03:41:03.204 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.205 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.205 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.205 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484581+0000", 2024-09-16T03:41:03.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484581+0000", 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484581+0000", 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1999999999999999e-07 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484597+0000", 2024-09-16T03:41:03.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6387000000000001e-05 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484607+0000", 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1989999999999995e-06 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484613+0000", 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.879e-06 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484707+0000", 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4692999999999996e-05 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484723+0000", 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5982999999999999e-05 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.497117+0000", 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012393665 2024-09-16T03:41:03.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.497169+0000", 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1647999999999997e-05 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.bs4 73/31 ECSubWrite(tid=93, reqid=osd.10.0:93, at_version=73'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.485256+0000", 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.698698952999999, 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012341438999999999, 2024-09-16T03:41:03.209 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485256+0000", 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485256+0000", 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485257+0000", 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.569999999999999e-07 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485274+0000", 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6991999999999999e-05 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485283+0000", 2024-09-16T03:41:03.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7110000000000007e-06 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485289+0000", 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8170000000000004e-06 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485346+0000", 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7258e-05 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485357+0000", 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0173999999999999e-05 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.497566+0000", 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012209108 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.497597+0000", 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1421999999999998e-05 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.213 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.bs4 73/31 ECSubWrite(tid=94, reqid=osd.10.0:94, at_version=73'42, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.486560+0000", 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.697394395, 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011849755, 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486560+0000", 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486560+0000", 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486561+0000", 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2399999999999998e-07 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486579+0000", 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7853e-05 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486589+0000", 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0108999999999999e-05 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486594+0000", 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1390000000000003e-06 2024-09-16T03:41:03.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.489863+0000", 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032689709999999999 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.489873+0000", 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0205e-05 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.498379+0000", 2024-09-16T03:41:03.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008506006 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.498410+0000", 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0848e-05 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.bs4 73/31 ECSubWrite(tid=95, reqid=client.4615.0:649, at_version=73'44, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.488312+0000", 2024-09-16T03:41:03.218 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.695642878999999, 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012984793, 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488312+0000", 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488312+0000", 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488312+0000", 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3300000000000003e-07 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488587+0000", 2024-09-16T03:41:03.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00027490799999999998 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488601+0000", 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3732e-05 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488610+0000", 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9590000000000001e-06 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493693+0000", 2024-09-16T03:41:03.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050829040000000001 2024-09-16T03:41:03.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493702+0000", 2024-09-16T03:41:03.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0850000000000003e-06 2024-09-16T03:41:03.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.501267+0000", 2024-09-16T03:41:03.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075654490000000001 2024-09-16T03:41:03.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.501297+0000", 2024-09-16T03:41:03.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9323e-05 2024-09-16T03:41:03.223 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.224 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.224 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:668 3.9s0 3:922516b8:::smithi12138290-20:head [append 0~458752] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.565260+0000", 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.618695079, 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011158611000000001, 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.225 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 668 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565260+0000", 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565260+0000", 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565260+0000", 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5499999999999997e-07 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565690+0000", 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042905800000000001 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565699+0000", 2024-09-16T03:41:03.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7340000000000003e-06 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565707+0000", 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0020000000000006e-06 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565762+0000", 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4660000000000002e-05 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565861+0000", 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8657000000000001e-05 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566603+0000", 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00074198500000000002 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.571732+0000", 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051299259999999999 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576358+0000", 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046256839999999997 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576418+0000", 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0050000000000003e-05 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.230 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:669 3.9s0 3:922516b8:::smithi12138290-20:head [append 0~81920] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:03.231 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.565724+0000", 2024-09-16T03:41:03.231 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.618230456999999, 2024-09-16T03:41:03.231 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010841539000000001, 2024-09-16T03:41:03.231 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.231 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.231 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.231 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.231 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.231 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.231 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 669 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565724+0000", 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565724+0000", 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565725+0000", 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5499999999999998e-07 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565805+0000", 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9880000000000001e-05 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565811+0000", 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1519999999999997e-06 2024-09-16T03:41:03.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565815+0000", 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7170000000000002e-06 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567047+0000", 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012322800000000001 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567135+0000", 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8120000000000003e-05 2024-09-16T03:41:03.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567651+0000", 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051601800000000003 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.571753+0000", 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041022810000000002 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576511+0000", 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047579620000000001 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576566+0000", 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4574000000000003e-05 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:670 3.9s0 3:922516b8:::smithi12138290-20:head [setxattr _header (57) in=7b] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.565823+0000", 2024-09-16T03:41:03.236 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.618131814, 2024-09-16T03:41:03.237 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012597641999999999, 2024-09-16T03:41:03.237 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.237 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.237 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.237 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.237 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.237 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.237 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 670 2024-09-16T03:41:03.237 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565823+0000", 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565823+0000", 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.238 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565824+0000", 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7100000000000001e-07 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565826+0000", 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565833+0000", 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0329999999999997e-06 2024-09-16T03:41:03.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565837+0000", 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9280000000000001e-06 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568016+0000", 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021785379999999998 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568074+0000", 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8239000000000003e-05 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568410+0000", 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00033627100000000002 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.571764+0000", 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033534540000000001 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578288+0000", 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065244969999999998 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578421+0000", 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013207 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.242 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:671 3.9s0 3:922516b8:::smithi12138290-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.565843+0000", 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.618111795000001, 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014185764999999999, 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 671 2024-09-16T03:41:03.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565843+0000", 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565843+0000", 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.244 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565843+0000", 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1100000000000002e-07 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565845+0000", 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6649999999999999e-06 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565849+0000", 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2039999999999999e-06 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565852+0000", 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0139999999999999e-06 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568727+0000", 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028753720000000002 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568777+0000", 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9438000000000003e-05 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578785+0000", 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010007771 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578870+0000", 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5657999999999999e-05 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.580029+0000", 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001158332 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:739 3.9s0 3:90188821:::smithi12138290-22:head [copy-from ver 76] snapc 1b=[1b,1a,19,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:56.475250+0000", 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.708704752, 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022129930999999999, 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.248 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 739 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475250+0000", 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475250+0000", 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475251+0000", 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5710000000000001e-06 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475259+0000", 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5290000000000003e-06 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475272+0000", 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2611e-05 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475279+0000", 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1790000000000002e-06 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475329+0000", 2024-09-16T03:41:03.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9895000000000003e-05 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475334+0000", 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4779999999999999e-06 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.477288+0000", 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019541319999999999 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.477384+0000", 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5963000000000003e-05 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.492225+0000", 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014841127000000001 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.494445+0000", 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022194129999999999 2024-09-16T03:41:03.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.497324+0000", 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028792800000000001 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.497380+0000", 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5753e-05 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:740 3.9s0 3:90188821:::smithi12138290-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:56.475294+0000", 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.7086607459999996, 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018563797, 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 740 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.255 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475294+0000", 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475294+0000", 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475294+0000", 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2599999999999998e-07 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475296+0000", 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.66e-06 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475302+0000", 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9410000000000004e-06 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475309+0000", 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4259999999999997e-06 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475377+0000", 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7725000000000006e-05 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475382+0000", 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2689999999999999e-06 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.477513+0000", 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00213043 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.477575+0000", 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1952999999999995e-05 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.493676+0000", 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016101790000000001 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.493762+0000", 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5504e-05 2024-09-16T03:41:03.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.493858+0000", 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5773000000000006e-05 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:760 3.9s0 3:9b8813dd:::smithi12138290-12:head [append 0~671744] snapc 1b=[1b,1a,19,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:56.539204+0000", 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.6447509220000001, 2024-09-16T03:41:03.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010315586999999999, 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 760 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.539204+0000", 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.539204+0000", 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.539205+0000", 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3030000000000001e-06 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.539921+0000", 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071606700000000003 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.539937+0000", 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5455e-05 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.539946+0000", 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9409999999999996e-06 2024-09-16T03:41:03.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.539997+0000", 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1591999999999997e-05 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.540090+0000", 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3185999999999999e-05 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.540695+0000", 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060461200000000003 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.549321+0000", 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086259660000000005 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.549465+0000", 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-09-16T03:41:03.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.549519+0000", 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4234999999999998e-05 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:763 3.9s0 3:9b8813dd:::smithi12138290-12:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,19,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:56.544451+0000", 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.6395032829999998, 2024-09-16T03:41:03.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010307274999999999, 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 763 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544451+0000", 2024-09-16T03:41:03.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544451+0000", 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544452+0000", 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4499999999999997e-07 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544456+0000", 2024-09-16T03:41:03.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2520000000000001e-06 2024-09-16T03:41:03.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544463+0000", 2024-09-16T03:41:03.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7920000000000004e-06 2024-09-16T03:41:03.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544468+0000", 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1660000000000002e-06 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.549829+0000", 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053609620000000004 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.549885+0000", 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5727999999999999e-05 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.550201+0000", 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00031615099999999998 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.550992+0000", 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00079084999999999995 2024-09-16T03:41:03.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.554658+0000", 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003666641 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.554759+0000", 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100188 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:764 3.9s0 3:9b8813dd:::smithi12138290-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:56.544476+0000", 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.6394785120000002, 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012277615, 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 764 2024-09-16T03:41:03.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544476+0000", 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544476+0000", 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544477+0000", 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1799999999999995e-07 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544480+0000", 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.016e-06 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544487+0000", 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2119999999999997e-06 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544492+0000", 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0509999999999996e-06 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.550597+0000", 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006104936 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.550679+0000", 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1734e-05 2024-09-16T03:41:03.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.555023+0000", 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043442339999999998 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.555108+0000", 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4781e-05 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.556754+0000", 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016461329999999999 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.cs3 78/31 ECSubWrite(tid=236, reqid=osd.0.0:236, at_version=78'124, trim_to=0'0, roll_forward_to=78'120))", 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:57.561063+0000", 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.6228918940000003, 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009692249, 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.278 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561063+0000", 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561063+0000", 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561063+0000", 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5700000000000002e-07 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561081+0000", 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7682000000000001e-05 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561090+0000", 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1260000000000004e-06 2024-09-16T03:41:03.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561104+0000", 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3867e-05 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561164+0000", 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0299e-05 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561171+0000", 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9299999999999997e-06 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.570716+0000", 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0095447669999999991 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.570755+0000", 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8921000000000002e-05 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:817 3.0s0 3:00cc7e13:::smithi12138290-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:58.596485+0000", 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.587470079, 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011026210999999999, 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 817 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596485+0000", 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596485+0000", 2024-09-16T03:41:03.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596485+0000", 2024-09-16T03:41:03.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6899999999999999e-07 2024-09-16T03:41:03.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596486+0000", 2024-09-16T03:41:03.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6050000000000001e-06 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596490+0000", 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.76e-06 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596494+0000", 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7859999999999998e-06 2024-09-16T03:41:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.599020+0000", 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025262689999999998 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.599114+0000", 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4060000000000004e-05 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.604963+0000", 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005848982 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.605056+0000", 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2462999999999999e-05 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.607511+0000", 2024-09-16T03:41:03.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024550169999999999 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:819 3.0s0 3:0353c118:::smithi12138290-10:head [read 0~1409024,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:58.596509+0000", 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.5874456949999995, 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096411810000000004, 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.289 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 819 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596509+0000", 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596509+0000", 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596509+0000", 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2600000000000001e-07 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596511+0000", 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5850000000000001e-06 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596514+0000", 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.145e-06 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596517+0000", 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9079999999999999e-06 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.599839+0000", 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003322583 2024-09-16T03:41:03.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.599937+0000", 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7424999999999996e-05 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.606150+0000", 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062133090000000002 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:820 3.0s0 3:0353c118:::smithi12138290-10:head [read 0~1409024,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:58.596520+0000", 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.587434944, 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010688052999999999, 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 820 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.294 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596520+0000", 2024-09-16T03:41:03.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596520+0000", 2024-09-16T03:41:03.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596520+0000", 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2100000000000001e-07 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596522+0000", 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.623e-06 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596525+0000", 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.18e-06 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.596528+0000", 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.785e-06 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.603132+0000", 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066047859999999996 2024-09-16T03:41:03.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.603193+0000", 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0717000000000002e-05 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:58.607208+0000", 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040147409999999996 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.298 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:03.300 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:41:03.301 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:03.298+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:03.400 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:03.398+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:03.407 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:03.408 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:03.408 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:41:03.408 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:03.416 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:41:03.480 INFO:tasks.rados.rados.0.smithi121.stdout:322: done (4 left) 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:323: copy_from oid 45 from oid 6 current snap is 28 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:319: finishing rollback tid 0 to smithi12138290-50 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:318: finishing rollback tid 0 to smithi12138290-9 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:318: finishing rollback tid 1 to smithi12138290-9 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:318: finishing rollback tid 2 to smithi12138290-9 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 9 v 71 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:319: finishing rollback tid 1 to smithi12138290-50 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:319: finishing rollback tid 2 to smithi12138290-50 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 50 v 69 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:316: finishing rollback tid 0 to smithi12138290-33 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:316: finishing rollback tid 1 to smithi12138290-33 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:316: finishing rollback tid 2 to smithi12138290-33 2024-09-16T03:41:03.481 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 33 v 35 (ObjNum 107 snap 20 seq_num 107) dirty exists 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: finishing write tid 1 to smithi12138290-14 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: finishing write tid 2 to smithi12138290-14 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: finishing write tid 3 to smithi12138290-14 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: finishing write tid 4 to smithi12138290-14 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: finishing write tid 5 to smithi12138290-14 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: finishing write tid 6 to smithi12138290-14 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: oid 14 updating version 0 to 152 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: oid 14 version 152 is already newer than 150 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: oid 14 updating version 152 to 154 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: oid 14 version 154 is already newer than 153 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: oid 14 version 154 is already newer than 151 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 14 v 154 (ObjNum 142 snap 28 seq_num 142) dirty exists 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:321: left oid 14 (ObjNum 142 snap 28 seq_num 142) 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:316: done (4 left) 2024-09-16T03:41:03.482 INFO:tasks.rados.rados.0.smithi121.stdout:318: done (3 left) 2024-09-16T03:41:03.483 INFO:tasks.rados.rados.0.smithi121.stdout:319: done (2 left) 2024-09-16T03:41:03.483 INFO:tasks.rados.rados.0.smithi121.stdout:321: done (1 left) 2024-09-16T03:41:03.483 INFO:tasks.rados.rados.0.smithi121.stdout:324: delete oid 23 current snap is 28 2024-09-16T03:41:03.495 INFO:tasks.rados.rados.0.smithi121.stdout:324: done (1 left) 2024-09-16T03:41:03.496 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 1 current snap is 28 2024-09-16T03:41:03.496 INFO:tasks.rados.rados.0.smithi121.stdout:325: seq_num 144 ranges {1949696=16384} 2024-09-16T03:41:03.501 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:03.498+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:03.507 INFO:tasks.rados.rados.0.smithi121.stdout:325: writing smithi12138290-1 from 1949696 to 1966080 tid 1 2024-09-16T03:41:03.507 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 12 current snap is 28 2024-09-16T03:41:03.507 INFO:tasks.rados.rados.0.smithi121.stdout:326: seq_num 145 ranges {3571712=32768} 2024-09-16T03:41:03.527 INFO:tasks.rados.rados.0.smithi121.stdout:326: writing smithi12138290-12 from 3571712 to 3604480 tid 1 2024-09-16T03:41:03.527 INFO:tasks.rados.rados.0.smithi121.stdout:327: snap_create 2024-09-16T03:41:03.533 INFO:tasks.rados.rados.0.smithi121.stdout:326: finishing write tid 1 to smithi12138290-12 2024-09-16T03:41:03.534 INFO:tasks.rados.rados.0.smithi121.stdout:325: finishing write tid 1 to smithi12138290-1 2024-09-16T03:41:03.535 INFO:tasks.rados.rados.0.smithi121.stdout:325: finishing write tid 2 to smithi12138290-1 2024-09-16T03:41:03.537 INFO:tasks.rados.rados.0.smithi121.stdout:325: finishing write tid 3 to smithi12138290-1 2024-09-16T03:41:03.537 INFO:tasks.rados.rados.0.smithi121.stdout:325: oid 1 updating version 0 to 164 2024-09-16T03:41:03.537 INFO:tasks.rados.rados.0.smithi121.stdout:325: oid 1 updating version 164 to 165 2024-09-16T03:41:03.537 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 1 v 165 (ObjNum 144 snap 28 seq_num 144) dirty exists 2024-09-16T03:41:03.537 INFO:tasks.rados.rados.0.smithi121.stdout:325: left oid 1 (ObjNum 144 snap 28 seq_num 144) 2024-09-16T03:41:03.538 INFO:tasks.rados.rados.0.smithi121.stdout:326: finishing write tid 2 to smithi12138290-12 2024-09-16T03:41:03.539 INFO:tasks.rados.rados.0.smithi121.stdout:326: finishing write tid 3 to smithi12138290-12 2024-09-16T03:41:03.539 INFO:tasks.rados.rados.0.smithi121.stdout:326: oid 12 updating version 0 to 83 2024-09-16T03:41:03.540 INFO:tasks.rados.rados.0.smithi121.stdout:326: oid 12 version 83 is already newer than 82 2024-09-16T03:41:03.540 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 83 (ObjNum 145 snap 28 seq_num 145) dirty exists 2024-09-16T03:41:03.540 INFO:tasks.rados.rados.0.smithi121.stdout:326: left oid 12 (ObjNum 145 snap 28 seq_num 145) 2024-09-16T03:41:03.571 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:03.572 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:03.572 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:41:03.572 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:41:03.572 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:03.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:41:03.602 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:03.600+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:03.604 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T03:41:03.602+0000 7f011120c640 -1 mon.a@0(leader).osd e84 definitely_dead 0 2024-09-16T03:41:03.604 INFO:teuthology.orchestra.run.smithi076.stderr:osd.12 is already down. 2024-09-16T03:41:03.618 INFO:tasks.thrashosds.thrasher:Removing osd 12, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15] 2024-09-16T03:41:03.618 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 12 2024-09-16T03:41:03.703 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:03.701+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:03.727 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:03.727 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:41:03.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:41:03.728 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:41:03.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.728 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:231 3.7s0 3:e441e0cd:::smithi12138290-34:head [append 0~737280] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:41:03.728 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.548164+0000", 2024-09-16T03:41:03.728 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.176059090000003, 2024-09-16T03:41:03.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074938289999999996, 2024-09-16T03:41:03.729 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.729 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.729 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.729 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.729 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.730 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.730 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 231 2024-09-16T03:41:03.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.730 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.548164+0000", 2024-09-16T03:41:03.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.548164+0000", 2024-09-16T03:41:03.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.548166+0000", 2024-09-16T03:41:03.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4929999999999999e-06 2024-09-16T03:41:03.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549023+0000", 2024-09-16T03:41:03.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00085696500000000001 2024-09-16T03:41:03.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549041+0000", 2024-09-16T03:41:03.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8612000000000001e-05 2024-09-16T03:41:03.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549051+0000", 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5389999999999996e-06 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549108+0000", 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6780999999999997e-05 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:41:03.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.549115+0000", 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3690000000000001e-06 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550938+0000", 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001823376 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.551048+0000", 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010907499999999999 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.552071+0000", 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0010230149999999999 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555488+0000", 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034173379999999998 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555573+0000", 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4932000000000006e-05 2024-09-16T03:41:03.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.555658+0000", 2024-09-16T03:41:03.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5334000000000007e-05 2024-09-16T03:41:03.739 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.739 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.739 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:234 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:17.550720+0000", 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.173503381000003, 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075177949999999999, 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 234 2024-09-16T03:41:03.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550720+0000", 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550720+0000", 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550721+0000", 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9200000000000001e-07 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550723+0000", 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2409999999999998e-06 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550730+0000", 2024-09-16T03:41:03.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8750000000000002e-06 2024-09-16T03:41:03.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.550736+0000", 2024-09-16T03:41:03.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8579999999999996e-06 2024-09-16T03:41:03.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556094+0000", 2024-09-16T03:41:03.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053587419999999997 2024-09-16T03:41:03.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:03.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.556171+0000", 2024-09-16T03:41:03.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7232999999999997e-05 2024-09-16T03:41:03.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.557335+0000", 2024-09-16T03:41:03.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011640000000000001 2024-09-16T03:41:03.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.557405+0000", 2024-09-16T03:41:03.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9979000000000002e-05 2024-09-16T03:41:03.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:17.558238+0000", 2024-09-16T03:41:03.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000832375 2024-09-16T03:41:03.746 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.746 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.746 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.746 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:264 3.7s0 3:e06e137e:::smithi12138290-27:head [copy-from ver 4] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.133546987000003, 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012313443, 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 264 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.747 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590679+0000", 2024-09-16T03:41:03.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1739999999999999e-06 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590686+0000", 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.43e-06 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590709+0000", 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2748999999999999e-05 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590718+0000", 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0359999999999995e-06 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590860+0000", 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000142208 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590966+0000", 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000106439 2024-09-16T03:41:03.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.599409+0000", 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008442583 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600331+0000", 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000922202 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.602907+0000", 2024-09-16T03:41:03.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002575787 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.602990+0000", 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2835000000000002e-05 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:265 3.7s0 3:e06e137e:::smithi12138290-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:41:03.752 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:41:03.753 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.133425148000001, 2024-09-16T03:41:03.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0092800720000000003, 2024-09-16T03:41:03.753 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.753 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.753 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.753 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.753 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.753 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 265 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.754 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590799+0000", 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590804+0000", 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3009999999999997e-06 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590832+0000", 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7851000000000002e-05 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590849+0000", 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6691e-05 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.591224+0000", 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037502100000000001 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.591256+0000", 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.184e-05 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.599957+0000", 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087011659999999998 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600004+0000", 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7218000000000001e-05 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600078+0000", 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4065000000000003e-05 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:296 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.253104829999998, 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084845249999999997, 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 296 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471120+0000", 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.804e-06 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471128+0000", 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0199999999999994e-06 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471144+0000", 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5957999999999999e-05 2024-09-16T03:41:03.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471153+0000", 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7299999999999994e-06 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471238+0000", 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4860999999999999e-05 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:41:03.762 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471244+0000", 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0859999999999998e-06 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.473493+0000", 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022493750000000001 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.473624+0000", 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130891 2024-09-16T03:41:03.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.764 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.764 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.764 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.479603+0000", 2024-09-16T03:41:03.764 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059788000000000003 2024-09-16T03:41:03.764 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.764 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.764 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:297 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.253048429000003, 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094313820000000003, 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.765 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 297 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.767 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:41:03.767 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9900000000000001e-07 2024-09-16T03:41:03.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.767 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.767 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471180+0000", 2024-09-16T03:41:03.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4619999999999998e-06 2024-09-16T03:41:03.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.768 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.768 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471186+0000", 2024-09-16T03:41:03.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3130000000000004e-06 2024-09-16T03:41:03.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.769 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471190+0000", 2024-09-16T03:41:03.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.194e-06 2024-09-16T03:41:03.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.770 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.770 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471314+0000", 2024-09-16T03:41:03.770 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012333299999999999 2024-09-16T03:41:03.770 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.770 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:03.771 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471322+0000", 2024-09-16T03:41:03.771 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8280000000000003e-06 2024-09-16T03:41:03.771 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.771 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.771 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.772 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.476742+0000", 2024-09-16T03:41:03.772 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054202740000000001 2024-09-16T03:41:03.772 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.772 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.772 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.772 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.476849+0000", 2024-09-16T03:41:03.773 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010663899999999999 2024-09-16T03:41:03.773 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.773 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.773 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.773 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480606+0000", 2024-09-16T03:41:03.773 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037578400000000001 2024-09-16T03:41:03.774 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.774 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.774 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.774 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.775 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.775 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:298 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:41:03.775 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:41:03.775 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.252999901999999, 2024-09-16T03:41:03.775 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011872347, 2024-09-16T03:41:03.775 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.776 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.776 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:03.776 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.776 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.776 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.776 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 298 2024-09-16T03:41:03.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.777 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.777 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.777 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.777 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:41:03.777 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.777 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.777 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2399999999999999e-07 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471227+0000", 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1149999999999998e-06 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.778 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.779 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.779 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471232+0000", 2024-09-16T03:41:03.779 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.152e-06 2024-09-16T03:41:03.779 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.779 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.779 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.779 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471237+0000", 2024-09-16T03:41:03.779 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9540000000000003e-06 2024-09-16T03:41:03.779 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.779 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471356+0000", 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011896900000000001 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471364+0000", 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480706+0000", 2024-09-16T03:41:03.780 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093421570000000002 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480820+0000", 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011411100000000001 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.483096+0000", 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022753819999999998 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:308 3.7s0 3:e441e0cd:::smithi12138290-34:head [create,setxattr h^blbcaNtSmmsQtZPdaRtPkhOOgN_Vi\\f`NtsZ[V (561) in=40b,setxattr T (17332) in=1b,setxattr UVoaZteYfmpthpcfl[gebaR\\TdqcNkhPRRdrgsX] (279) in=40b,setxattr bTPgfXb\\YXtXkQag\\Qh`hlXr[posd]]cje]^eebl (1121) in=40b,setxattr hm^gakWSOOOfNOTph^tma]iaNsaqfjqZYb\\seZNW (3297) in=40b,setxattr YiNn[]m_q`YlQlaV^mRn^ (17332) in=21b,setxattr \\tllpkXThmWXSkOopogrdtjY]f^a^sem_krafjfi (947) in=40b,setxattr RXiRZRQVSaUTaZmQT^bfqZhS^mZsqadaPdW]ijm` (1963) in=40b,setxattr WTmsTZhNpU`tZY\\^aROaNdmtYsjjsgWWboehPOP` (19508) in=40b,setxattr UZmWZeWaoiX]jmbn][UadnNkcZc_hdcjidtNsRYo (1999) in=40b,setxattr NSebjRYYQ`SbaatVjZWNab`cenkljRq`^]Xr\\jRm (11788) in=40b,setxattr ip`daUaop[rdbNPo\\^c[Wcelsl]a][RScQZ[kfa^ (18524) in=40b,setxattr qU_qcNcQpNkegs_Od`jieTXX[mto]lsR^jTcYprR (593) in=40b,setxattr dboVg`srq[bReagdR]fX\\Pe`Warrp]\\[XpSipTN[ (1683) in=40b,setxattr ZdTaVj[ScjsOiQ]a_]R[RQWoYohnjd`fi[dYmfs` (840) in=40b,setxattr N_[blmbpstWhtUgW]ZjZ[\\O_erqcnQN]R_V^U\\RT (1015) in=40b,setxattr jXbP_mNsadnq`UVfmc^lfZUs\\r_[ZnrahqTUk^Zp (383) in=40b,setxattr nmemd\\ipehj\\q^UVY_YtSrhZjcabNXhlai_noQSO (807) in=40b,setxattr Rbc\\cahhlcrckUioieUbQnrseNkZkdPf\\rUmNb]S (384) in=40b,setxattr ZiOkXqg^PobTdZgrtZ]]l^Rap_tfSQoUQqaefj]] (69) in=40b] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.233738973999998, 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013488215, 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.782 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 308 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.783 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490486+0000", 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.601e-06 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490741+0000", 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00025543300000000001 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490756+0000", 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4949e-05 2024-09-16T03:41:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490764+0000", 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1659999999999997e-06 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.492684+0000", 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019205509999999999 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.785 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.493193+0000", 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050880499999999998 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.494675+0000", 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001482443 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.496347+0000", 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001671358 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.786 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.503822+0000", 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074755639999999996 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.503973+0000", 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000150345 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.787 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.cs4 73/31 ECSubWrite(tid=183, reqid=osd.0.0:183, at_version=73'90, trim_to=0'0, roll_forward_to=73'86))", 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.506500+0000", 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.217723122000001, 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069504069999999996, 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.788 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.506500+0000", 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.506500+0000", 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.506501+0000", 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.789 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.506518+0000", 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7075000000000002e-05 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.506529+0000", 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0971999999999999e-05 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.506536+0000", 2024-09-16T03:41:03.790 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9029999999999998e-06 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.506590+0000", 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4027000000000003e-05 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.506597+0000", 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.917e-06 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.791 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.513416+0000", 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068190239999999999 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.513451+0000", 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4938999999999999e-05 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.9s3 73/55 ECSubWrite(tid=95, reqid=client.4615.0:668, at_version=73'49, trim_to=0'0, roll_forward_to=73'47))", 2024-09-16T03:41:03.792 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.566610+0000", 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.157613771999999, 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0095467829999999997, 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566610+0000", 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.793 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566610+0000", 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566610+0000", 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3099999999999997e-07 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566665+0000", 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4790000000000002e-05 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566678+0000", 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3098999999999999e-05 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566700+0000", 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2004000000000001e-05 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566773+0000", 2024-09-16T03:41:03.795 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3108999999999994e-05 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566785+0000", 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1325000000000001e-05 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576121+0000", 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093363079999999998 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.796 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576156+0000", 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5417000000000003e-05 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.9s3 73/55 ECSubWrite(tid=96, reqid=client.4615.0:669, at_version=73'50, trim_to=0'0, roll_forward_to=73'47))", 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.567667+0000", 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.156556177000001, 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085155620000000008, 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.797 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567667+0000", 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567667+0000", 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.798 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567668+0000", 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9999999999999997e-07 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567697+0000", 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8731e-05 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567710+0000", 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3053e-05 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.799 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567718+0000", 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2109999999999998e-06 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568977+0000", 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012587410000000001 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568987+0000", 2024-09-16T03:41:03.800 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0122000000000001e-05 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576161+0000", 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007174003 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576183+0000", 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2001000000000002e-05 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.ds2 73/31 ECSubWrite(tid=122, reqid=client.4615.0:673, at_version=73'59, trim_to=0'0, roll_forward_to=73'56))", 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.568320+0000", 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.155903707, 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078391590000000001, 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568320+0000", 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.802 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568320+0000", 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568320+0000", 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4099999999999998e-07 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.803 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568346+0000", 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5446999999999999e-05 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568359+0000", 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3149000000000001e-05 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568368+0000", 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7069999999999998e-06 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.804 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568422+0000", 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4537999999999999e-05 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568429+0000", 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4160000000000001e-06 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576123+0000", 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007694083 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576159+0000", 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6078000000000003e-05 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.9s3 73/55 ECSubWrite(tid=97, reqid=client.4615.0:670, at_version=73'51, trim_to=0'0, roll_forward_to=73'47))", 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.568386+0000", 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.155837498, 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0097616679999999994, 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.806 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568386+0000", 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568386+0000", 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568387+0000", 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5000000000000002e-07 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568403+0000", 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6155999999999999e-05 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568421+0000", 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7779999999999999e-05 2024-09-16T03:41:03.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568427+0000", 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0569999999999998e-06 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569278+0000", 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00085171900000000002 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.809 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569284+0000", 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578114+0000", 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00883046 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578148+0000", 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3469000000000001e-05 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.ds2 73/31 ECSubWrite(tid=123, reqid=client.4615.0:674, at_version=73'60, trim_to=0'0, roll_forward_to=73'56))", 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.568889+0000", 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.155334152, 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072899030000000004, 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568889+0000", 2024-09-16T03:41:03.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568889+0000", 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568890+0000", 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568906+0000", 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5637999999999998e-05 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568916+0000", 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0117e-05 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568923+0000", 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1369999999999996e-06 2024-09-16T03:41:03.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569175+0000", 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000252289 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569180+0000", 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1109999999999999e-06 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576163+0000", 2024-09-16T03:41:03.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069823200000000002 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576179+0000", 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6605000000000001e-05 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:675 3.7s0 3:e441e0cd:::smithi12138290-34:head [rmxattr NSebjRYYQ`SbaatVjZWNab`cenkljRq`^]Xr\\jRm in=40b,rmxattr RXiRZRQVSaUTaZmQT^bfqZhS^mZsqadaPdW]ijm` in=40b,rmxattr UZmWZeWaoiX]jmbn][UadnNkcZc_hdcjidtNsRYo in=40b,rmxattr ZdTaVj[ScjsOiQ]a_]R[RQWoYohnjd`fi[dYmfs` in=40b,rmxattr bTPgfXb\\YXtXkQag\\Qh`hlXr[posd]]cje]^eebl in=40b,rmxattr dboVg`srq[bReagdR]fX\\Pe`Warrp]\\[XpSipTN[ in=40b,rmxattr h^blbcaNtSmmsQtZPdaRtPkhOOgN_Vi\\f`NtsZ[V in=40b,rmxattr ip`daUaop[rdbNPo\\^c[Wcelsl]a][RScQZ[kfa^ in=40b] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.569765+0000", 2024-09-16T03:41:03.815 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.154458879, 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087892790000000005, 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 675 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569765+0000", 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569765+0000", 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569766+0000", 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.435e-06 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569773+0000", 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8519999999999998e-06 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569784+0000", 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1236999999999999e-05 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569790+0000", 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5210000000000002e-06 2024-09-16T03:41:03.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569882+0000", 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2668000000000001e-05 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569975+0000", 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2733999999999997e-05 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.570353+0000", 2024-09-16T03:41:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037774 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578125+0000", 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077719399999999998 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578181+0000", 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5958999999999997e-05 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.821 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578554+0000", 2024-09-16T03:41:03.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037319300000000002 2024-09-16T03:41:03.821 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.821 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.821 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.821 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s4 78/31 ECSubWrite(tid=141, reqid=osd.6.0:141, at_version=78'121, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:03.821 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:57.559137+0000", 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1650864199999997, 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087583680000000007, 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559137+0000", 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559137+0000", 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559138+0000", 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0900000000000001e-07 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559154+0000", 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6388999999999998e-05 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559165+0000", 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0986e-05 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559171+0000", 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9270000000000002e-06 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.560571+0000", 2024-09-16T03:41:03.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014004079999999999 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.560580+0000", 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7490000000000004e-06 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.567862+0000", 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072813729999999998 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.825 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.567895+0000", 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3827e-05 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.cs4 78/31 ECSubWrite(tid=236, reqid=osd.0.0:236, at_version=78'124, trim_to=0'0, roll_forward_to=78'120))", 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:57.561161+0000", 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1630622730000004, 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094978739999999999, 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.826 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561161+0000", 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561161+0000", 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561165+0000", 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6490000000000001e-06 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561187+0000", 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2005999999999999e-05 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561201+0000", 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3726e-05 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561211+0000", 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0088000000000001e-05 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561257+0000", 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6690999999999997e-05 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561264+0000", 2024-09-16T03:41:03.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8179999999999998e-06 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.570617+0000", 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093528239999999992 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.570659+0000", 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2072e-05 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:803 3.7s0 3:e441e0cd:::smithi12138290-34:head [rollback 19] snapc 1b=[1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:57.732821+0000", 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.9914021680000005, 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072185929999999997, 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 803 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.831 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732821+0000", 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732821+0000", 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732822+0000", 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3500000000000002e-07 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732824+0000", 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0930000000000001e-06 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732829+0000", 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8069999999999996e-06 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732833+0000", 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2169999999999999e-06 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.734123+0000", 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001290072 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.734197+0000", 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4367000000000001e-05 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.734808+0000", 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061112099999999995 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.739920+0000", 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051119989999999999 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.739977+0000", 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.694e-05 2024-09-16T03:41:03.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.740040+0000", 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2642000000000001e-05 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:804 3.7s0 3:e441e0cd:::smithi12138290-34:head [append 0~0] snapc 1b=[1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:57.732855+0000", 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.9913679999999996, 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0081578229999999998, 2024-09-16T03:41:03.836 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 804 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732855+0000", 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732855+0000", 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732856+0000", 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-07 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732858+0000", 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5790000000000002e-06 2024-09-16T03:41:03.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732866+0000", 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.204e-06 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.732871+0000", 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3249999999999998e-06 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.735532+0000", 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026612939999999998 2024-09-16T03:41:03.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.735609+0000", 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6672e-05 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.735960+0000", 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00035132199999999999 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.740043+0000", 2024-09-16T03:41:03.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040825000000000002 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.740952+0000", 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00090927900000000003 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.741013+0000", 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1168999999999997e-05 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es5 84/55 ECSubWrite(tid=51, reqid=osd.3.0:51, at_version=84'73, trim_to=0'0, roll_forward_to=83'69))", 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:03.485593+0000", 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.238630655, 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077874040000000004, 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485593+0000", 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485593+0000", 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485594+0000", 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.283e-06 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485616+0000", 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1491000000000001e-05 2024-09-16T03:41:03.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485633+0000", 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7248000000000002e-05 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485642+0000", 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3309999999999993e-06 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485840+0000", 2024-09-16T03:41:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000198123 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485850+0000", 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0142e-05 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.493346+0000", 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074953320000000004 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:03.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.493380+0000", 2024-09-16T03:41:03.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4453999999999999e-05 2024-09-16T03:41:03.846 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.846 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.846 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.846 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:03.846 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:03.846 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:03.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:41:03.848 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:03.845+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:03.913 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:41:03.948 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T03:41:03.946+0000 7f011120c640 -1 mon.a@0(leader).osd e84 definitely_dead 0 2024-09-16T03:41:03.948 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:03.946+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:03.959 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:03.959 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:03.959 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:41:03.959 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:03.968 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:41:04.033 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:41:04.034 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:41:04.051 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:41:04.051 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:41:04.051 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:04.050+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:04.067 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:41:04.068 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:41:04.084 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:41:04.085 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:41:04.100 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:41:04.100 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:41:04.112 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:04.112 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:04.116 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:41:04.116 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:41:04.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:41:04.133 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:41:04.134 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:41:04.150 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:41:04.151 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:41:04.151 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:04.149+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:04.166 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:41:04.166 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:41:04.183 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:41:04.183 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:41:04.199 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:41:04.200 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:41:04.216 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:41:04.217 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:41:04.217 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-16T03:41:04.233 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:41:04.234 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:41:04.251 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:04.250+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:04.251 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:41:04.251 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:41:04.266 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:04.266 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:04.268 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:41:04.268 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:41:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:41:04.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:41:04.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:41:04.352 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:04.350+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:04.378 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:04.378 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:04.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:41:04.452 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:04.451+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:04.478 INFO:teuthology.orchestra.run.smithi076.stderr:marked out osd.12. 2024-09-16T03:41:04.481 INFO:tasks.rados.rados.0.smithi121.stdout:325: done (3 left) 2024-09-16T03:41:04.481 INFO:tasks.rados.rados.0.smithi121.stdout:326: done (2 left) 2024-09-16T03:41:04.481 INFO:tasks.rados.rados.0.smithi121.stdout:327: done (1 left) 2024-09-16T03:41:04.481 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 39 current snap is 29 2024-09-16T03:41:04.482 INFO:tasks.rados.rados.0.smithi121.stdout:328: seq_num 146 ranges {1703936=49152} 2024-09-16T03:41:04.491 INFO:tasks.rados.rados.0.smithi121.stdout:328: writing smithi12138290-39 from 1703936 to 1753088 tid 1 2024-09-16T03:41:04.491 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 49 current snap is 29 2024-09-16T03:41:04.491 INFO:tasks.rados.rados.0.smithi121.stdout:329: seq_num 147 ranges {163840=65536} 2024-09-16T03:41:04.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:41:04.493 INFO:tasks.rados.rados.0.smithi121.stdout:329: writing smithi12138290-49 from 163840 to 229376 tid 1 2024-09-16T03:41:04.493 INFO:tasks.rados.rados.0.smithi121.stdout:330: read oid 15 snap -1 2024-09-16T03:41:04.493 INFO:tasks.rados.rados.0.smithi121.stdout:330: expect (ObjNum 56 snap 0 seq_num 56) 2024-09-16T03:41:04.493 INFO:tasks.rados.rados.0.smithi121.stdout:331: snap_create 2024-09-16T03:41:04.494 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.12 2024-09-16T03:41:04.494 INFO:tasks.thrashosds.thrasher:remote for osd 12 is ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:41:04.495 DEBUG:teuthology.orchestra.run.smithi076:> type ceph-objectstore-tool 2024-09-16T03:41:04.495 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:04.495 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:04.496 INFO:tasks.rados.rados.0.smithi121.stdout:328: finishing write tid 1 to smithi12138290-39 2024-09-16T03:41:04.496 INFO:tasks.rados.rados.0.smithi121.stdout:328: finishing write tid 2 to smithi12138290-39 2024-09-16T03:41:04.496 INFO:tasks.rados.rados.0.smithi121.stdout:328: finishing write tid 3 to smithi12138290-39 2024-09-16T03:41:04.496 INFO:tasks.rados.rados.0.smithi121.stdout:328: oid 39 updating version 0 to 85 2024-09-16T03:41:04.496 INFO:tasks.rados.rados.0.smithi121.stdout:328: oid 39 version 85 is already newer than 84 2024-09-16T03:41:04.496 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 39 v 85 (ObjNum 146 snap 29 seq_num 146) dirty exists 2024-09-16T03:41:04.496 INFO:tasks.rados.rados.0.smithi121.stdout:328: left oid 39 (ObjNum 146 snap 29 seq_num 146) 2024-09-16T03:41:04.501 INFO:tasks.rados.rados.0.smithi121.stdout:330: expect (ObjNum 56 snap 0 seq_num 56) 2024-09-16T03:41:04.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:41:04.549 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T03:41:04.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-12 --journal-path /var/lib/ceph/osd/ceph-12/journal --op list-pgs 2024-09-16T03:41:04.553 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:41:04.552+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:41:04.607 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:04.607 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:04.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:41:04.655 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:04.653+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:04.719 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:04.719 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:04.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:41:04.756 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:04.754+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:04.830 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:04.830 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:04.839 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:41:04.884 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:41:04.882+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:41:04.941 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:04.942 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:04.950 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:41:04.957 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:04.956+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:05.058 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:05.056+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:05.095 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:05.095 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:05.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:41:05.160 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:05.158+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:05.206 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:05.206 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:05.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:41:05.261 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:05.259+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:05.318 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:05.318 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:05.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:41:05.361 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:05.360+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:05.444 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:05.445 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:05.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:41:05.462 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:05.460+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:05.479 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T03:41:05.485 INFO:tasks.rados.rados.0.smithi121.stdout:328: done (4 left) 2024-09-16T03:41:05.485 INFO:tasks.rados.rados.0.smithi121.stdout:330: done (3 left) 2024-09-16T03:41:05.485 INFO:tasks.rados.rados.0.smithi121.stdout:331: done (2 left) 2024-09-16T03:41:05.485 INFO:tasks.rados.rados.0.smithi121.stdout:332: setattr oid 24 current snap is 30 2024-09-16T03:41:05.488 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 30 current snap is 30 2024-09-16T03:41:05.489 INFO:tasks.rados.rados.0.smithi121.stdout:333: seq_num 148 ranges {7618560=81920} 2024-09-16T03:41:05.533 INFO:tasks.rados.rados.0.smithi121.stdout:333: writing smithi12138290-30 from 7618560 to 7700480 tid 1 2024-09-16T03:41:05.533 INFO:tasks.rados.rados.0.smithi121.stdout:334: delete oid 19 current snap is 30 2024-09-16T03:41:05.533 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 24 v 79 (ObjNum 137 snap 27 seq_num 137) dirty exists 2024-09-16T03:41:05.533 INFO:tasks.rados.rados.0.smithi121.stdout:329: finishing write tid 1 to smithi12138290-49 2024-09-16T03:41:05.533 INFO:tasks.rados.rados.0.smithi121.stdout:329: finishing write tid 2 to smithi12138290-49 2024-09-16T03:41:05.533 INFO:tasks.rados.rados.0.smithi121.stdout:329: finishing write tid 3 to smithi12138290-49 2024-09-16T03:41:05.533 INFO:tasks.rados.rados.0.smithi121.stdout:329: oid 49 updating version 0 to 36 2024-09-16T03:41:05.534 INFO:tasks.rados.rados.0.smithi121.stdout:329: oid 49 version 36 is already newer than 35 2024-09-16T03:41:05.534 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 49 v 36 (ObjNum 147 snap 29 seq_num 147) dirty exists 2024-09-16T03:41:05.534 INFO:tasks.rados.rados.0.smithi121.stdout:329: left oid 49 (ObjNum 147 snap 29 seq_num 147) 2024-09-16T03:41:05.538 INFO:tasks.rados.rados.0.smithi121.stdout:329: done (4 left) 2024-09-16T03:41:05.538 INFO:tasks.rados.rados.0.smithi121.stdout:332: done (3 left) 2024-09-16T03:41:05.539 INFO:tasks.rados.rados.0.smithi121.stdout:334: done (2 left) 2024-09-16T03:41:05.539 INFO:tasks.rados.rados.0.smithi121.stdout:335: snap_create 2024-09-16T03:41:05.539 INFO:tasks.rados.rados.0.smithi121.stdout:333: finishing write tid 1 to smithi12138290-30 2024-09-16T03:41:05.539 INFO:tasks.rados.rados.0.smithi121.stdout:333: finishing write tid 2 to smithi12138290-30 2024-09-16T03:41:05.540 INFO:tasks.rados.rados.0.smithi121.stdout:333: finishing write tid 3 to smithi12138290-30 2024-09-16T03:41:05.541 INFO:tasks.rados.rados.0.smithi121.stdout:333: oid 30 updating version 0 to 168 2024-09-16T03:41:05.541 INFO:tasks.rados.rados.0.smithi121.stdout:333: oid 30 version 168 is already newer than 167 2024-09-16T03:41:05.541 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 30 v 168 (ObjNum 148 snap 30 seq_num 148) dirty exists 2024-09-16T03:41:05.541 INFO:tasks.rados.rados.0.smithi121.stdout:333: left oid 30 (ObjNum 148 snap 30 seq_num 148) 2024-09-16T03:41:05.563 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:05.562+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:05.571 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:05.572 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:05.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:41:05.664 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:05.663+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:05.723 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:05.723 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:05.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:41:05.775 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:05.774+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:05.835 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:05.836 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:05.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:41:05.866 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:05.864+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:05.967 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:05.966+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:05.986 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:05.986 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:05.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:41:06.068 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:06.067+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:06.138 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:06.138 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:06.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:41:06.169 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:06.168+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:06.250 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:06.250 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:06.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:41:06.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:41:06.303 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:06.301+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:06.362 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:06.362 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:06.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:41:06.371 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:06.369+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:06.394 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:41:06.395 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:41:06.412 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:41:06.412 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:41:06.429 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:41:06.429 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:41:06.445 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:41:06.445 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:41:06.461 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:41:06.461 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:41:06.472 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:06.470+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:06.478 INFO:tasks.rados.rados.0.smithi121.stdout:333: done (2 left) 2024-09-16T03:41:06.478 INFO:tasks.rados.rados.0.smithi121.stdout:335: done (1 left) 2024-09-16T03:41:06.478 INFO:tasks.rados.rados.0.smithi121.stdout:336: delete oid 3 current snap is 31 2024-09-16T03:41:06.487 INFO:tasks.rados.rados.0.smithi121.stdout:336: done (1 left) 2024-09-16T03:41:06.487 INFO:tasks.rados.rados.0.smithi121.stdout:337: rollback oid 26 current snap is 31 2024-09-16T03:41:06.487 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 26 to 29 2024-09-16T03:41:06.487 INFO:tasks.rados.rados.0.smithi121.stdout:338: rollback oid 17 current snap is 31 2024-09-16T03:41:06.488 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 17 to 24 2024-09-16T03:41:06.488 INFO:tasks.rados.rados.0.smithi121.stdout:339: read oid 35 snap 15 2024-09-16T03:41:06.488 INFO:tasks.rados.rados.0.smithi121.stdout:339: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:41:06.488 INFO:tasks.rados.rados.0.smithi121.stdout:340: delete oid 41 current snap is 31 2024-09-16T03:41:06.488 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:41:06.488 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:41:06.494 INFO:tasks.rados.rados.0.smithi121.stdout:339: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:41:06.510 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:41:06.511 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:41:06.523 INFO:tasks.rados.rados.0.smithi121.stdout:339: done (4 left) 2024-09-16T03:41:06.523 INFO:tasks.rados.rados.0.smithi121.stdout:340: done (3 left) 2024-09-16T03:41:06.523 INFO:tasks.rados.rados.0.smithi121.stdout:341: rollback oid 9 current snap is 31 2024-09-16T03:41:06.523 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 9 to 28 2024-09-16T03:41:06.523 INFO:tasks.rados.rados.0.smithi121.stdout:323: finishing copy_from racing read to smithi12138290-45 2024-09-16T03:41:06.523 INFO:tasks.rados.rados.0.smithi121.stdout:337: finishing rollback tid 0 to smithi12138290-26 2024-09-16T03:41:06.523 INFO:tasks.rados.rados.0.smithi121.stdout:338: finishing rollback tid 0 to smithi12138290-17 2024-09-16T03:41:06.523 INFO:tasks.rados.rados.0.smithi121.stdout:323: finishing copy_from to smithi12138290-45 2024-09-16T03:41:06.524 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 45 v 8 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-09-16T03:41:06.524 INFO:tasks.rados.rados.0.smithi121.stdout:323: done (3 left) 2024-09-16T03:41:06.524 INFO:tasks.rados.rados.0.smithi121.stdout:342: read oid 28 snap -1 2024-09-16T03:41:06.524 INFO:tasks.rados.rados.0.smithi121.stdout:342: expect (ObjNum 73 snap 6 seq_num 73) 2024-09-16T03:41:06.524 INFO:tasks.rados.rados.0.smithi121.stdout:343: setattr oid 43 current snap is 31 2024-09-16T03:41:06.526 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 22 current snap is 31 2024-09-16T03:41:06.526 INFO:tasks.rados.rados.0.smithi121.stdout:344: seq_num 152 ranges {1916928=147456} 2024-09-16T03:41:06.532 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:41:06.532 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:41:06.533 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:06.533 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:06.536 INFO:teuthology.orchestra.run.smithi076.stdout:3.ds0 2024-09-16T03:41:06.537 INFO:teuthology.orchestra.run.smithi076.stdout:3.cs2 2024-09-16T03:41:06.537 INFO:teuthology.orchestra.run.smithi076.stdout:3.as0 2024-09-16T03:41:06.537 INFO:teuthology.orchestra.run.smithi076.stdout:3.4s1 2024-09-16T03:41:06.537 INFO:teuthology.orchestra.run.smithi076.stdout:3.3s6 2024-09-16T03:41:06.538 INFO:tasks.rados.rados.0.smithi121.stdout:344: writing smithi12138290-22 from 1916928 to 2064384 tid 1 2024-09-16T03:41:06.538 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 34 current snap is 31 2024-09-16T03:41:06.538 INFO:tasks.rados.rados.0.smithi121.stdout:341: finishing rollback tid 0 to smithi12138290-9 2024-09-16T03:41:06.538 INFO:tasks.rados.rados.0.smithi121.stdout:345: seq_num 153 ranges {3407872=163840} 2024-09-16T03:41:06.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:41:06.553 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:41:06.558 INFO:tasks.rados.rados.0.smithi121.stdout:345: writing smithi12138290-34 from 3407872 to 3571712 tid 1 2024-09-16T03:41:06.558 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 11 current snap is 31 2024-09-16T03:41:06.558 INFO:tasks.rados.rados.0.smithi121.stdout:346: seq_num 154 ranges {1982464=180224} 2024-09-16T03:41:06.570 INFO:tasks.rados.rados.0.smithi121.stdout:346: writing smithi12138290-11 from 1982464 to 2162688 tid 1 2024-09-16T03:41:06.570 INFO:tasks.rados.rados.0.smithi121.stdout:347: read oid 5 snap -1 2024-09-16T03:41:06.570 INFO:tasks.rados.rados.0.smithi121.stdout:347: expect (ObjNum 135 snap 27 seq_num 135) 2024-09-16T03:41:06.570 INFO:tasks.rados.rados.0.smithi121.stdout:348: setattr oid 39 current snap is 31 2024-09-16T03:41:06.570 INFO:tasks.rados.rados.0.smithi121.stdout:341: finishing rollback tid 1 to smithi12138290-9 2024-09-16T03:41:06.570 INFO:tasks.rados.rados.0.smithi121.stdout:341: finishing rollback tid 2 to smithi12138290-9 2024-09-16T03:41:06.570 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 9 v 83 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:41:06.570 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 43 v 162 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:41:06.570 INFO:tasks.rados.rados.0.smithi121.stdout:342: expect (ObjNum 73 snap 6 seq_num 73) 2024-09-16T03:41:06.572 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:41:06.572 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:41:06.572 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:06.570+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:06.590 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:41:06.590 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:41:06.607 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:41:06.607 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:41:06.607 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-16T03:41:06.625 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:41:06.626 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:41:06.626 INFO:tasks.rados.rados.0.smithi121.stdout:344: finishing write tid 1 to smithi12138290-22 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:344: finishing write tid 2 to smithi12138290-22 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:344: finishing write tid 3 to smithi12138290-22 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:344: oid 22 updating version 0 to 85 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:344: oid 22 version 85 is already newer than 84 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 22 v 85 (ObjNum 152 snap 31 seq_num 152) dirty exists 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:344: left oid 22 (ObjNum 152 snap 31 seq_num 152) 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:345: finishing write tid 1 to smithi12138290-34 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:345: finishing write tid 2 to smithi12138290-34 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:345: finishing write tid 3 to smithi12138290-34 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:345: oid 34 updating version 0 to 61 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:345: oid 34 version 61 is already newer than 60 2024-09-16T03:41:06.627 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 34 v 61 (ObjNum 153 snap 31 seq_num 153) dirty exists 2024-09-16T03:41:06.628 INFO:tasks.rados.rados.0.smithi121.stdout:345: left oid 34 (ObjNum 153 snap 31 seq_num 153) 2024-09-16T03:41:06.628 INFO:tasks.rados.rados.0.smithi121.stdout:346: finishing write tid 1 to smithi12138290-11 2024-09-16T03:41:06.628 INFO:tasks.rados.rados.0.smithi121.stdout:346: finishing write tid 2 to smithi12138290-11 2024-09-16T03:41:06.628 INFO:tasks.rados.rados.0.smithi121.stdout:346: finishing write tid 3 to smithi12138290-11 2024-09-16T03:41:06.628 INFO:tasks.rados.rados.0.smithi121.stdout:346: oid 11 updating version 0 to 159 2024-09-16T03:41:06.628 INFO:tasks.rados.rados.0.smithi121.stdout:346: oid 11 version 159 is already newer than 158 2024-09-16T03:41:06.628 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 11 v 159 (ObjNum 154 snap 31 seq_num 154) dirty exists 2024-09-16T03:41:06.628 INFO:tasks.rados.rados.0.smithi121.stdout:346: left oid 11 (ObjNum 154 snap 31 seq_num 154) 2024-09-16T03:41:06.628 INFO:tasks.rados.rados.0.smithi121.stdout:347: expect (ObjNum 135 snap 27 seq_num 135) 2024-09-16T03:41:06.643 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:41:06.643 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:41:06.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:41:06.642+0000 7f447dd9b640 0 -- 172.21.15.76:0/2054724714 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f448021c040 msgr2=0x7f448021e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:41:06.656 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:06.656 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:06.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:41:06.673 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:41:06.672+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:41:06.754 INFO:tasks.rados.rados.0.smithi121.stdout:337: finishing rollback tid 1 to smithi12138290-26 2024-09-16T03:41:06.754 INFO:tasks.rados.rados.0.smithi121.stdout:341: done (9 left) 2024-09-16T03:41:06.754 INFO:tasks.rados.rados.0.smithi121.stdout:342: done (8 left) 2024-09-16T03:41:06.754 INFO:tasks.rados.rados.0.smithi121.stdout:343: done (7 left) 2024-09-16T03:41:06.754 INFO:tasks.rados.rados.0.smithi121.stdout:344: done (6 left) 2024-09-16T03:41:06.754 INFO:tasks.rados.rados.0.smithi121.stdout:345: done (5 left) 2024-09-16T03:41:06.755 INFO:tasks.rados.rados.0.smithi121.stdout:346: done (4 left) 2024-09-16T03:41:06.755 INFO:tasks.rados.rados.0.smithi121.stdout:347: done (3 left) 2024-09-16T03:41:06.755 INFO:tasks.rados.rados.0.smithi121.stdout:349: snap_create 2024-09-16T03:41:06.755 INFO:tasks.rados.rados.0.smithi121.stdout:337: finishing rollback tid 2 to smithi12138290-26 2024-09-16T03:41:06.755 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 26 v 164 (ObjNum 138 snap 27 seq_num 138) dirty exists 2024-09-16T03:41:06.755 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 39 v 87 (ObjNum 146 snap 29 seq_num 146) dirty exists 2024-09-16T03:41:06.755 INFO:tasks.rados.rados.0.smithi121.stdout:338: finishing rollback tid 1 to smithi12138290-17 2024-09-16T03:41:06.755 INFO:tasks.rados.rados.0.smithi121.stdout:338: finishing rollback tid 2 to smithi12138290-17 2024-09-16T03:41:06.755 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 17 v 166 (ObjNum 118 snap 24 seq_num 118) dirty exists 2024-09-16T03:41:06.774 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:06.773+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:06.809 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:06.809 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:06.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:41:06.860 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:41:06.861 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:41:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:41:06.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:41:06.875 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:06.874+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:06.921 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:06.921 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:41:06.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:41:06.921 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:41:06.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.921 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:375 3.3s0 3:c98c73f3:::smithi12138290-40:head [append 0~671744] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:06.921 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.381409556999998, 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086953299999999994, 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 375 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537291+0000", 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.59e-06 2024-09-16T03:41:06.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537708+0000", 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041628699999999999 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537721+0000", 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3738e-05 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537728+0000", 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4409999999999998e-06 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537798+0000", 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0543000000000005e-05 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537931+0000", 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013217700000000001 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.539173+0000", 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012421660000000001 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.540584+0000", 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001411673 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545889+0000", 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053041340000000003 2024-09-16T03:41:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545985+0000", 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6581000000000002e-05 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:376 3.3s0 3:c98c73f3:::smithi12138290-40:head [append 0~770048] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.376693295999999, 2024-09-16T03:41:06.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087274550000000003, 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 376 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:41:06.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542008+0000", 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.756e-06 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542428+0000", 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041977300000000002 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542440+0000", 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2894000000000001e-05 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542447+0000", 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.584e-06 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542503+0000", 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5483999999999999e-05 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542630+0000", 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012754500000000001 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.543864+0000", 2024-09-16T03:41:06.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012343090000000001 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545743+0000", 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018785799999999999 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550638+0000", 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048951790000000004 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550733+0000", 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5351000000000002e-05 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:377 3.3s0 3:c98c73f3:::smithi12138290-40:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.376237228000001, 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084428420000000007, 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.933 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 377 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542463+0000", 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2600000000000002e-07 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542467+0000", 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6040000000000003e-06 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542473+0000", 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6019999999999999e-06 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542476+0000", 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.393e-06 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.544778+0000", 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023019329999999999 2024-09-16T03:41:06.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.544876+0000", 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7448000000000006e-05 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545337+0000", 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046103399999999998 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.547435+0000", 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020983320000000001 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550831+0000", 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033956920000000001 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550905+0000", 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4078000000000005e-05 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:378 3.3s0 3:c98c73f3:::smithi12138290-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.376200806, 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011059941, 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 378 2024-09-16T03:41:06.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1699999999999998e-07 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542502+0000", 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8040000000000002e-06 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542509+0000", 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3320000000000003e-06 2024-09-16T03:41:06.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542515+0000", 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.49e-06 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.546046+0000", 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035316670000000001 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:06.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.546144+0000", 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8040000000000003e-05 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.551270+0000", 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005125597 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.551384+0000", 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011363599999999999 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.553558+0000", 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021747580000000002 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.433892686, 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060036960000000002, 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.944 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0699999999999997e-07 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484830+0000", 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3292999999999999e-05 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484841+0000", 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0891e-05 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484848+0000", 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8070000000000002e-06 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484897+0000", 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8887999999999997e-05 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484904+0000", 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8430000000000004e-06 2024-09-16T03:41:06.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490773+0000", 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058694940000000003 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490810+0000", 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6973e-05 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=93, reqid=osd.10.0:93, at_version=73'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.433109351000001, 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056680849999999998, 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5599999999999999e-07 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485609+0000", 2024-09-16T03:41:06.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9238000000000001e-05 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485620+0000", 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0781000000000001e-05 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485630+0000", 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7270000000000002e-06 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485684+0000", 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4517e-05 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485695+0000", 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0542e-05 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491225+0000", 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055304880000000001 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491258+0000", 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2635999999999997e-05 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=94, reqid=osd.10.0:94, at_version=73'42, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.431921573, 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065321210000000001, 2024-09-16T03:41:06.953 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:41:06.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0399999999999995e-07 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486805+0000", 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6245999999999999e-05 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486815+0000", 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0239e-05 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486820+0000", 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3020000000000002e-06 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488642+0000", 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001822001 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488651+0000", 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5639999999999997e-06 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493274+0000", 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046231290000000001 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493310+0000", 2024-09-16T03:41:06.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5936000000000003e-05 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=95, reqid=client.4615.0:649, at_version=73'44, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.430206511, 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097259600000000005, 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488494+0000", 2024-09-16T03:41:06.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5099999999999998e-07 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488724+0000", 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000229764 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488736+0000", 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.221e-05 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488744+0000", 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3110000000000006e-06 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492660+0000", 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039159759999999998 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492669+0000", 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3449999999999995e-06 2024-09-16T03:41:06.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498200+0000", 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055302169999999996 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498219+0000", 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9185999999999999e-05 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=97, reqid=client.4615.0:650, at_version=73'45, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.425757608, 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052967650000000002, 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0399999999999996e-07 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493161+0000", 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021878300000000001 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493171+0000", 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5929999999999995e-06 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493178+0000", 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2389999999999996e-06 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493367+0000", 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000189581 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493389+0000", 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1389000000000001e-05 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498222+0000", 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048334980000000003 2024-09-16T03:41:06.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498238+0000", 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6178e-05 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s4 73/55 ECSubWrite(tid=98, reqid=osd.10.0:98, at_version=73'24, trim_to=0'0, roll_forward_to=56'20))", 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.425034626, 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0063600940000000002, 2024-09-16T03:41:06.967 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493680+0000", 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4543e-05 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493688+0000", 2024-09-16T03:41:06.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.687e-06 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493694+0000", 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8209999999999998e-06 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493766+0000", 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2169999999999995e-05 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493776+0000", 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2890000000000003e-06 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499989+0000", 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062135269999999999 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500025+0000", 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5596999999999998e-05 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.971 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=99, reqid=client.4615.0:651, at_version=73'46, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.494942+0000", 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.423757227999999, 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052685420000000002, 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494942+0000", 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494942+0000", 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494943+0000", 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2200000000000003e-07 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495356+0000", 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041339099999999999 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495371+0000", 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4537e-05 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495379+0000", 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6829999999999997e-06 2024-09-16T03:41:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495467+0000", 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7392000000000006e-05 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495476+0000", 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5240000000000006e-06 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500158+0000", 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046818190000000003 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500211+0000", 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2473999999999999e-05 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=100, reqid=client.4615.0:652, at_version=73'47, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:41:06.976 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.421626975000001, 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083283569999999998, 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:41:06.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497073+0000", 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.61e-07 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497223+0000", 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014987799999999999 2024-09-16T03:41:06.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497236+0000", 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3278999999999999e-05 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497245+0000", 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8130000000000002e-06 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497329+0000", 2024-09-16T03:41:06.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3671999999999994e-05 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497338+0000", 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7000000000000003e-06 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.505360+0000", 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080216100000000002 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.505401+0000", 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0744000000000002e-05 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=93, reqid=osd.14.0:92, at_version=0'0, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.418561834, 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0079817680000000002, 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.981 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7400000000000002e-07 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500148+0000", 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037999999999999e-05 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500163+0000", 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5109999999999999e-05 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500172+0000", 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8419999999999994e-06 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500224+0000", 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1759999999999999e-05 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500233+0000", 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8359999999999998e-06 2024-09-16T03:41:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508092+0000", 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078594029999999992 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508119+0000", 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7005e-05 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=101, reqid=client.4615.0:653, at_version=73'48, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.500387+0000", 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.418312114999999, 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077870200000000004, 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500387+0000", 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500387+0000", 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500388+0000", 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500412+0000", 2024-09-16T03:41:06.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4386e-05 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500421+0000", 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1020000000000003e-06 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500433+0000", 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2011000000000001e-05 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500496+0000", 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500507+0000", 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0499000000000001e-05 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508155+0000", 2024-09-16T03:41:06.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076484200000000004 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508174+0000", 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9011000000000001e-05 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=94, reqid=osd.14.0:94, at_version=73'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:41:06.990 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.417577516, 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070195409999999998, 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:41:06.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6499999999999999e-07 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501143+0000", 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0565e-05 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501156+0000", 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3278999999999999e-05 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501164+0000", 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0900000000000005e-06 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501227+0000", 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2682999999999996e-05 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501245+0000", 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8108e-05 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508122+0000", 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068766180000000001 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508141+0000", 2024-09-16T03:41:06.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9533000000000001e-05 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=102, reqid=osd.10.0:102, at_version=73'50, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.416768325, 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087707529999999992, 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:06.995 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:06.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501932+0000", 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3e-07 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501949+0000", 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7711e-05 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501962+0000", 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2549e-05 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501969+0000", 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1720000000000001e-06 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502019+0000", 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0340000000000003e-05 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502028+0000", 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8570000000000001e-06 2024-09-16T03:41:06.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510672+0000", 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086435399999999999 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510702+0000", 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9954000000000002e-05 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:06.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s4 73/55 ECSubWrite(tid=103, reqid=osd.10.0:98, at_version=0'0, trim_to=0'0, roll_forward_to=73'24))", 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502008+0000", 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.416691053999999, 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006105933, 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502008+0000", 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:07.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502008+0000", 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502009+0000", 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:07.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502016+0000", 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3660000000000003e-06 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502024+0000", 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.542e-06 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502030+0000", 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9379999999999997e-06 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502199+0000", 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016938200000000001 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502209+0000", 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508072+0000", 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058631170000000002 2024-09-16T03:41:07.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508114+0000", 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2546999999999998e-05 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=95, reqid=osd.14.0:95, at_version=73'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.416613555, 2024-09-16T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085979139999999999, 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:07.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3099999999999998e-07 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502103+0000", 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6769000000000002e-05 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502114+0000", 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1258e-05 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502121+0000", 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6429999999999998e-06 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:07.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502175+0000", 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3698e-05 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502183+0000", 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6589999999999996e-06 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510650+0000", 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084665680000000007 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510684+0000", 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3887999999999998e-05 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.es6 78/55 ECSubWrite(tid=38, reqid=osd.3.0:38, at_version=78'57, trim_to=0'0, roll_forward_to=76'53))", 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555743+0000", 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3629558920000004, 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058697580000000001, 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:07.009 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555743+0000", 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555743+0000", 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555744+0000", 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003e-06 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555812+0000", 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7830000000000006e-05 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555841+0000", 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8619000000000002e-05 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555854+0000", 2024-09-16T03:41:07.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3424000000000001e-05 2024-09-16T03:41:07.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:07.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556254+0000", 2024-09-16T03:41:07.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039962800000000002 2024-09-16T03:41:07.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556264+0000", 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6020000000000006e-06 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561560+0000", 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052968650000000004 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561613+0000", 2024-09-16T03:41:07.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2787000000000001e-05 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 78/31 ECSubWrite(tid=120, reqid=osd.14.0:120, at_version=78'105, trim_to=0'0, roll_forward_to=77'101))", 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.556185+0000", 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3625140059999996, 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054648550000000002, 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:07.014 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556185+0000", 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556185+0000", 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556186+0000", 2024-09-16T03:41:07.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:41:07.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:07.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556210+0000", 2024-09-16T03:41:07.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3618999999999999e-05 2024-09-16T03:41:07.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:07.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556220+0000", 2024-09-16T03:41:07.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0699e-05 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556229+0000", 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0739999999999993e-06 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556310+0000", 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1716000000000002e-05 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556316+0000", 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561619+0000", 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053029849999999996 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561650+0000", 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1433999999999998e-05 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:07.018 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:07.019 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:07.019 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:07.019 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:07.019 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:07.020 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:41:07.021 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:07.018+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:07.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-12 --journal-path /var/lib/ceph/osd/ceph-12/journal --op export --pgid 3.cs2 --file /var/log/ceph/exp.3.cs2.12 2024-09-16T03:41:07.121 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:07.119+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:07.125 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:07.126 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:07.126 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:41:07.126 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:07.134 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:41:07.222 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:41:07.220+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:41:07.278 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:07.278 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:07.278 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:41:07.278 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:41:07.278 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:07.286 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:41:07.323 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:41:07.321+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:41:07.390 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:07.391 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:41:07.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:41:07.391 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-09-16T03:41:07.391 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:07.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:41:07.424 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:07.422+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:07.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:41:07.504 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:07.505 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:07.505 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:41:07.505 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:07.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:41:07.525 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:07.523+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:07.589 INFO:tasks.rados.rados.0.smithi121.stdout:337: done (3 left) 2024-09-16T03:41:07.589 INFO:tasks.rados.rados.0.smithi121.stdout:338: done (2 left) 2024-09-16T03:41:07.589 INFO:tasks.rados.rados.0.smithi121.stdout:348: done (1 left) 2024-09-16T03:41:07.589 INFO:tasks.rados.rados.0.smithi121.stdout:349: done (0 left) 2024-09-16T03:41:07.589 INFO:tasks.rados.rados.0.smithi121.stdout:350: copy_from oid 12 from oid 20 current snap is 32 2024-09-16T03:41:07.589 INFO:tasks.rados.rados.0.smithi121.stdout:351: snap_create 2024-09-16T03:41:07.603 INFO:tasks.rados.rados.0.smithi121.stdout:350: finishing copy_from racing read to smithi12138290-12 2024-09-16T03:41:07.606 INFO:tasks.rados.rados.0.smithi121.stdout:350: finishing copy_from to smithi12138290-12 2024-09-16T03:41:07.607 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 51 (ObjNum 119 snap 24 seq_num 119) dirty exists 2024-09-16T03:41:07.629 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:07.627+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:07.665 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:07.665 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:07.665 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:41:07.665 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:41:07.665 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:07.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:41:07.729 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:41:07.727+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:41:07.780 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:07.780 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:41:07.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:41:07.780 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:41:07.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.780 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:228 3.fs0 3:fe9db9b8:::smithi12138290-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.236175369999998, 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011210417, 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 228 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.781 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540949+0000", 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0399999999999996e-07 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540952+0000", 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6860000000000002e-06 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540960+0000", 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9729999999999997e-06 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540966+0000", 2024-09-16T03:41:07.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7509999999999997e-06 2024-09-16T03:41:07.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541041+0000", 2024-09-16T03:41:07.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5216999999999995e-05 2024-09-16T03:41:07.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.541046+0000", 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0000000000000004e-06 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544875+0000", 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038287339999999999 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.544964+0000", 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9245000000000004e-05 2024-09-16T03:41:07.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550540+0000", 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055763649999999998 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.550618+0000", 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7908e-05 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.552159+0000", 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015409340000000001 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:345 3.fs0 3:fe9db9b8:::smithi12138290-24:head [copy-from ver 4] snapc d=[d,b,a,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:31.497360+0000", 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.279764188999998, 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013644073, 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:07.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 345 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497360+0000", 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497360+0000", 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497362+0000", 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.598e-06 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497371+0000", 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0180000000000007e-06 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497382+0000", 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1562e-05 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497389+0000", 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4409999999999998e-06 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497446+0000", 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7054e-05 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497451+0000", 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.339e-06 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499316+0000", 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001864517 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499415+0000", 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9077e-05 2024-09-16T03:41:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.505946+0000", 2024-09-16T03:41:07.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006531294 2024-09-16T03:41:07.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.507163+0000", 2024-09-16T03:41:07.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001216649 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.510932+0000", 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037689820000000001 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.511004+0000", 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2541999999999998e-05 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:346 3.fs0 3:fe9db9b8:::smithi12138290-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.279719679000003, 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096185339999999998, 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 346 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.794 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497405+0000", 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0499999999999999e-07 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497407+0000", 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497412+0000", 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3859999999999998e-06 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497415+0000", 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1439999999999999e-06 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497492+0000", 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6577999999999998e-05 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.497497+0000", 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.023e-06 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499594+0000", 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020968380000000002 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.499632+0000", 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8133000000000002e-05 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.506768+0000", 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071363449999999997 2024-09-16T03:41:07.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.506883+0000", 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011502000000000001 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:31.507023+0000", 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013991399999999999 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:383 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.234819762999997, 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010993826999999999, 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 383 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.800 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542305+0000", 2024-09-16T03:41:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542308+0000", 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6450000000000001e-06 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542313+0000", 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2970000000000003e-06 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.542317+0000", 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5979999999999999e-06 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.543899+0000", 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015818539999999999 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.544026+0000", 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012688099999999999 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.553298+0000", 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092721520000000005 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:383 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.226734456999999, 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010309801, 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.804 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 383 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550390+0000", 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.77e-07 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550392+0000", 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0490000000000002e-06 2024-09-16T03:41:07.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550397+0000", 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9699999999999998e-06 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550401+0000", 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4039999999999999e-06 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.553954+0000", 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003552955 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554020+0000", 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6389000000000004e-05 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.560700+0000", 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066796570000000003 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:384 3.fs0 3:fe9db9b8:::smithi12138290-24:b [read 0~2834432,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.226719166999999, 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011709803, 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 384 2024-09-16T03:41:07.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550405+0000", 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7399999999999999e-07 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550408+0000", 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.526e-06 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550412+0000", 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.408e-06 2024-09-16T03:41:07.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.550415+0000", 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8260000000000001e-06 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554733+0000", 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004318087 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.554837+0000", 2024-09-16T03:41:07.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010340699999999999 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.562115+0000", 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072782749999999998 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 73/31 ECSubWrite(tid=91, reqid=osd.14.0:91, at_version=73'73, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.482643+0000", 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.294480807999999, 2024-09-16T03:41:07.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010037870000000001, 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482643+0000", 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482643+0000", 2024-09-16T03:41:07.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482644+0000", 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.43e-07 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482667+0000", 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3268000000000002e-05 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482677+0000", 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0488e-05 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482686+0000", 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6999999999999997e-06 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482741+0000", 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4625e-05 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482746+0000", 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6250000000000004e-06 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.492656+0000", 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099094620000000008 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.492681+0000", 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5559e-05 2024-09-16T03:41:07.817 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s5 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.483373+0000", 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.293751575, 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012019056, 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483373+0000", 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483373+0000", 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483373+0000", 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9900000000000001e-07 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483385+0000", 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1656e-05 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483392+0000", 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2640000000000002e-06 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483398+0000", 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4160000000000001e-06 2024-09-16T03:41:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487424+0000", 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040257720000000004 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487432+0000", 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2029999999999992e-06 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.495353+0000", 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079208460000000005 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.495392+0000", 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000001e-05 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.8s5 74/31 ECSubWrite(tid=210, reqid=osd.11.0:210, at_version=74'23, trim_to=0'0, roll_forward_to=73'21))", 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.530435+0000", 2024-09-16T03:41:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.246688773000001, 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097168529999999993, 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.530435+0000", 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.530435+0000", 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.530436+0000", 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6e-07 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.530456+0000", 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0117999999999999e-05 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.530467+0000", 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1362000000000001e-05 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.530477+0000", 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5650000000000007e-06 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.530624+0000", 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014719900000000001 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.530637+0000", 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2585999999999999e-05 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.540115+0000", 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0094781099999999997 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.540152+0000", 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7653e-05 2024-09-16T03:41:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:829 3.fs0 3:fe9db9b8:::smithi12138290-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:58.611255+0000", 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1658693759999998, 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098423030000000002, 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 829 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.611255+0000", 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.611255+0000", 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.611255+0000", 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1399999999999997e-07 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.611257+0000", 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0779999999999998e-06 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.611262+0000", 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.758e-06 2024-09-16T03:41:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.611265+0000", 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0180000000000002e-06 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.615246+0000", 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039808350000000003 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.615342+0000", 2024-09-16T03:41:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5551999999999996e-05 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.619783+0000", 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044418540000000003 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.619881+0000", 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7182999999999994e-05 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:58.621097+0000", 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001216611 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:877 3.fs0 3:f06c201b:::smithi12138290-9:head [rollback b] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:02.481100+0000", 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.2960245209999997, 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097508270000000001, 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 877 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481100+0000", 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481100+0000", 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481100+0000", 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8999999999999997e-07 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481111+0000", 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1223000000000001e-05 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481116+0000", 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5179999999999998e-06 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481120+0000", 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3660000000000002e-06 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481170+0000", 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0136999999999998e-05 2024-09-16T03:41:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481175+0000", 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3320000000000002e-06 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.484756+0000", 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035815439999999999 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.484839+0000", 2024-09-16T03:41:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2400999999999999e-05 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.485351+0000", 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051251200000000002 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.486603+0000", 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012516179999999999 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.490753+0000", 2024-09-16T03:41:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041501339999999998 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.490851+0000", 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7552000000000004e-05 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:878 3.fs0 3:f06c201b:::smithi12138290-9:head [append 0~0] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:02.481126+0000", 2024-09-16T03:41:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.2959986700000004, 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098421510000000004, 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 878 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481126+0000", 2024-09-16T03:41:07.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481126+0000", 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481126+0000", 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.96e-07 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481127+0000", 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3540000000000001e-06 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481131+0000", 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4070000000000002e-06 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481137+0000", 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1109999999999996e-06 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481206+0000", 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9628000000000005e-05 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.481213+0000", 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2210000000000002e-06 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.485645+0000", 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044322889999999998 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.485735+0000", 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0410999999999999e-05 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.486072+0000", 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033708000000000002 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.489548+0000", 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003475578 2024-09-16T03:41:07.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.490926+0000", 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013774550000000001 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:02.490968+0000", 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2221000000000001e-05 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:890 3.cs0 3:3aac74e6:::smithi12138290-45:head [copy-from ver 8] snapc 1d=[1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.479937+0000", 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.2971876140000003, 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0206764810000002, 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 890 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.845 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479937+0000", 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479937+0000", 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479938+0000", 2024-09-16T03:41:07.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.449e-06 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479944+0000", 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.716e-06 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479955+0000", 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0978999999999999e-05 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479962+0000", 2024-09-16T03:41:07.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6490000000000008e-06 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480163+0000", 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00020048200000000001 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480165+0000", 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5150000000000001e-06 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:04.484569+0000", 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0044037020000001 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:04.484573+0000", 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.393e-06 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:05.485339+0000", 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0007665370000001 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:05.485343+0000", 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1280000000000001e-06 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.489225+0000", 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.003881829 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.489381+0000", 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015567200000000001 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.497074+0000", 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076932750000000003 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.498962+0000", 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018883190000000001 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.500193+0000", 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012302280000000001 2024-09-16T03:41:07.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.500613+0000", 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000420608 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:891 3.cs0 3:3aac74e6:::smithi12138290-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.479980+0000", 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.297144436, 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0183411609999999, 2024-09-16T03:41:07.852 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 891 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479980+0000", 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479980+0000", 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479980+0000", 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.44e-07 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479983+0000", 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.537e-06 2024-09-16T03:41:07.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479990+0000", 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479996+0000", 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2660000000000001e-06 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480191+0000", 2024-09-16T03:41:07.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019537600000000001 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480192+0000", 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:04.484794+0000", 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.004602634 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:04.484796+0000", 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1780000000000002e-06 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:05.485533+0000", 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0007368139999999 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:05.485535+0000", 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1610000000000001e-06 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.490391+0000", 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.004855499 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.490447+0000", 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6635000000000002e-05 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.497865+0000", 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0074178259999999998 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.497970+0000", 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010501900000000001 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.498321+0000", 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000350699 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.859 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:917 3.cs0 3:38c596e1:::smithi12138290-26:head [append 0~0] snapc 20=[20,1f,1e,1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:06.486696+0000", 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.290428669, 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012181173999999999, 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 917 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486696+0000", 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486696+0000", 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486697+0000", 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2979999999999999e-06 2024-09-16T03:41:07.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486701+0000", 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.031e-06 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486712+0000", 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1348999999999999e-05 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486718+0000", 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8189999999999997e-06 2024-09-16T03:41:07.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486780+0000", 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1487000000000003e-05 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486783+0000", 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.422e-06 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.490500+0000", 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037167010000000002 2024-09-16T03:41:07.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.490744+0000", 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024434800000000003 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.491343+0000", 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00059870300000000002 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.494393+0000", 2024-09-16T03:41:07.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030504109999999998 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.498636+0000", 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004243124 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.498877+0000", 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024048099999999999 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:918 3.cs0 3:38c596e1:::smithi12138290-26:head [rollback 1f] snapc 20=[20,1f,1e,1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:06.486733+0000", 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.2903914379999999, 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.131424705, 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 918 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.866 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486733+0000", 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486733+0000", 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486733+0000", 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9200000000000001e-07 2024-09-16T03:41:07.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486739+0000", 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2259999999999996e-06 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486745+0000", 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8530000000000003e-06 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486751+0000", 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.293e-06 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486831+0000", 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0649999999999995e-05 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486833+0000", 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0499999999999999e-06 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.492045+0000", 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052117079999999998 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.492197+0000", 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015166499999999999 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for degraded object", 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.492401+0000", 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000204184 2024-09-16T03:41:07.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.615507+0000", 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123105987 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.615617+0000", 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011032 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.616101+0000", 2024-09-16T03:41:07.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00048422100000000001 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.617760+0000", 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016584150000000001 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.617965+0000", 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000205114 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.618158+0000", 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019252700000000001 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:919 3.cs0 3:38c596e1:::smithi12138290-26:head [append 0~0] snapc 20=[20,1f,1e,1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:06.486760+0000", 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.290364507, 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.13308576799999999, 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.873 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 919 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486760+0000", 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486760+0000", 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486760+0000", 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1699999999999999e-07 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486762+0000", 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6500000000000001e-06 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486767+0000", 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8500000000000002e-06 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486772+0000", 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1120000000000004e-06 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486875+0000", 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103686 2024-09-16T03:41:07.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486876+0000", 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.097e-06 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.492468+0000", 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005591347 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for degraded object", 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.492650+0000", 2024-09-16T03:41:07.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000181826 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.616512+0000", 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123862271 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.616635+0000", 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012274000000000001 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.617024+0000", 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038954899999999999 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.617761+0000", 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073695399999999995 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.619655+0000", 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001894171 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.619846+0000", 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019019800000000001 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:920 3.cs0 3:32badfe7:::smithi12138290-17:head [append 0~0] snapc 20=[20,1f,1e,1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:06.486779+0000", 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.2903454599999999, 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012841195999999999, 2024-09-16T03:41:07.880 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 920 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486779+0000", 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486779+0000", 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486779+0000", 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4000000000000002e-07 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486785+0000", 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7309999999999998e-06 2024-09-16T03:41:07.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486792+0000", 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6359999999999997e-06 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486797+0000", 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1139999999999997e-06 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486918+0000", 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121202 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486919+0000", 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.252e-06 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.492679+0000", 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057600150000000003 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.492832+0000", 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015310599999999999 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.493353+0000", 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00052116900000000004 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.494430+0000", 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010764909999999999 2024-09-16T03:41:07.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.499167+0000", 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047371599999999998 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.499620+0000", 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045288000000000001 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:921 3.cs0 3:32badfe7:::smithi12138290-17:head [rollback 1a] snapc 20=[20,1f,1e,1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:06.486815+0000", 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.2903090420000001, 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.14829077099999999, 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 921 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486815+0000", 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486815+0000", 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486816+0000", 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5600000000000001e-07 2024-09-16T03:41:07.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486818+0000", 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2350000000000002e-06 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486826+0000", 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0169999999999996e-06 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486831+0000", 2024-09-16T03:41:07.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1429999999999997e-06 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486956+0000", 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012463799999999999 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486957+0000", 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.004e-06 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.493938+0000", 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069816289999999996 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.494104+0000", 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000165916 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for degraded object", 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.494275+0000", 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017115300000000001 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.632631+0000", 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.13835546300000001 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.632737+0000", 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000105874 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.633218+0000", 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00048122799999999999 2024-09-16T03:41:07.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.634738+0000", 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015200509999999999 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.634929+0000", 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019085400000000001 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.635106+0000", 2024-09-16T03:41:07.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017710999999999999 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:922 3.cs0 3:32badfe7:::smithi12138290-17:head [append 0~0] snapc 20=[20,1f,1e,1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:06.486838+0000", 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.2902861699999999, 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15121583099999999, 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:07.894 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 922 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486838+0000", 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:07.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486838+0000", 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486839+0000", 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.15e-07 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486841+0000", 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1639999999999999e-06 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486847+0000", 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.409e-06 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486853+0000", 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6910000000000002e-06 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486995+0000", 2024-09-16T03:41:07.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000141843 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.486997+0000", 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.891e-06 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.494447+0000", 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0074507560000000002 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for degraded object", 2024-09-16T03:41:07.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.494542+0000", 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4924000000000001e-05 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.633675+0000", 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.13913323599999999 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.633768+0000", 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2347000000000002e-05 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.634135+0000", 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00036748 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.634739+0000", 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00060383300000000002 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.637863+0000", 2024-09-16T03:41:07.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031241540000000001 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:06.638054+0000", 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000190688 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:07.901 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:07.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:41:07.903 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:07.901+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:08.004 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:41:08.002+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:41:08.009 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:08.009 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:08.009 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:41:08.009 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:08.017 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:41:08.105 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:08.103+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:08.121 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:08.122 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:08.122 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:41:08.122 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:41:08.122 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:08.130 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:41:08.206 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:08.204+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:08.235 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:41:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:41:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:41:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 73/55 ECSubWrite(tid=24, reqid=osd.3.0:24, at_version=73'39, trim_to=0'0, roll_forward_to=72'35))", 2024-09-16T03:41:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.478776+0000", 2024-09-16T03:41:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.753885125, 2024-09-16T03:41:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085502010000000003, 2024-09-16T03:41:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478776+0000", 2024-09-16T03:41:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478776+0000", 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478776+0000", 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7199999999999999e-07 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478796+0000", 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9762000000000002e-05 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478807+0000", 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0368e-05 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478814+0000", 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5379999999999997e-06 2024-09-16T03:41:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478866+0000", 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1960999999999999e-05 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478875+0000", 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3600000000000002e-06 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487290+0000", 2024-09-16T03:41:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008414932 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487326+0000", 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5707999999999998e-05 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.0s2 73/31 ECSubWrite(tid=85, reqid=osd.1.0:85, at_version=73'40, trim_to=0'0, roll_forward_to=72'36))", 2024-09-16T03:41:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.479171+0000", 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.753489794, 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080606229999999994, 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479171+0000", 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479171+0000", 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479172+0000", 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7899999999999998e-07 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479191+0000", 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8802999999999999e-05 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479202+0000", 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0865e-05 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479209+0000", 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6809999999999998e-06 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479259+0000", 2024-09-16T03:41:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.922e-05 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.479269+0000", 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0017000000000001e-05 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487209+0000", 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079406809999999998 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487232+0000", 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2677e-05 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 73/31 ECSubWrite(tid=110, reqid=osd.6.0:110, at_version=73'89, trim_to=0'0, roll_forward_to=72'85))", 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.480970+0000", 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.751691427000001, 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089551219999999994, 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480970+0000", 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480970+0000", 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480971+0000", 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8400000000000003e-07 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480991+0000", 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0577999999999998e-05 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481005+0000", 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4369000000000001e-05 2024-09-16T03:41:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481014+0000", 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481088+0000", 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4051e-05 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481098+0000", 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7820000000000005e-06 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489895+0000", 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087973840000000001 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489925+0000", 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9388000000000001e-05 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds1 73/31 ECSubWrite(tid=119, reqid=osd.12.0:119, at_version=73'56, trim_to=0'0, roll_forward_to=73'54))", 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.481659+0000", 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.751002468999999, 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082272739999999997, 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481659+0000", 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481659+0000", 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481659+0000", 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1699999999999998e-07 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481677+0000", 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7518999999999999e-05 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481686+0000", 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3340000000000008e-06 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481694+0000", 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6450000000000005e-06 2024-09-16T03:41:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481750+0000", 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6385000000000003e-05 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481760+0000", 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.842e-06 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489853+0000", 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080927240000000008 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489886+0000", 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3308e-05 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s5 73/31 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=73'31, trim_to=0'0, roll_forward_to=66'29))", 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.484484+0000", 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.748177329000001, 2024-09-16T03:41:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085472250000000003, 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484484+0000", 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484484+0000", 2024-09-16T03:41:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484484+0000", 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8200000000000001e-07 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484507+0000", 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2535999999999999e-05 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484517+0000", 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.084e-05 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484537+0000", 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9174e-05 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484602+0000", 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5584000000000001e-05 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484611+0000", 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1339999999999993e-06 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.493014+0000", 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084031120000000008 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.493031+0000", 2024-09-16T03:41:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6463000000000001e-05 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 73/31 ECSubWrite(tid=94, reqid=osd.10.0:94, at_version=73'42, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.487044+0000", 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.745617522, 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084560920000000001, 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487044+0000", 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487044+0000", 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487044+0000", 2024-09-16T03:41:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8899999999999999e-07 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487072+0000", 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7889999999999999e-05 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487082+0000", 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0098999999999999e-05 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487089+0000", 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2409999999999998e-06 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487733+0000", 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00064390699999999999 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.487758+0000", 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4159999999999999e-05 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.495464+0000", 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077063560000000001 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.495500+0000", 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5750000000000002e-05 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.7s1 74/55 ECSubWrite(tid=62, reqid=osd.9.0:61, at_version=0'0, trim_to=0'0, roll_forward_to=74'44))", 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.533331+0000", 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.699330065, 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084000640000000005, 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533331+0000", 2024-09-16T03:41:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533331+0000", 2024-09-16T03:41:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533332+0000", 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.61e-07 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533342+0000", 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0210000000000001e-05 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533352+0000", 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0195e-05 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533361+0000", 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1749999999999994e-06 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533419+0000", 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8029000000000003e-05 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533429+0000", 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.541696+0000", 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082673150000000008 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.541731+0000", 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4938000000000003e-05 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 74/55 ECSubWrite(tid=27, reqid=client.4615.0:683, at_version=74'42, trim_to=0'0, roll_forward_to=73'39))", 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.533459+0000", 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.699201768, 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085407710000000008, 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533459+0000", 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533459+0000", 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533460+0000", 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533481+0000", 2024-09-16T03:41:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0571999999999999e-05 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533493+0000", 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2116999999999999e-05 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533500+0000", 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6290000000000001e-06 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533588+0000", 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7546000000000005e-05 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533600+0000", 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1812e-05 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.541974+0000", 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083744969999999998 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542000+0000", 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5837999999999999e-05 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds1 74/31 ECSubWrite(tid=126, reqid=client.4615.0:679, at_version=74'62, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.533660+0000", 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.699000592000001, 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083896070000000003, 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533660+0000", 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533660+0000", 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533661+0000", 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533680+0000", 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8777e-05 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533691+0000", 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1354e-05 2024-09-16T03:41:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533699+0000", 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7870000000000002e-06 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533777+0000", 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7498999999999998e-05 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533787+0000", 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542029+0000", 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082426200000000008 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542050+0000", 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0886e-05 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.bs5 74/31 ECSubWrite(tid=106, reqid=osd.10.0:106, at_version=74'52, trim_to=0'0, roll_forward_to=73'50))", 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.534506+0000", 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.698154761, 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090150509999999996, 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534506+0000", 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534506+0000", 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534507+0000", 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.91e-07 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534542+0000", 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5018999999999998e-05 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534551+0000", 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6889999999999999e-06 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534560+0000", 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3040000000000005e-06 2024-09-16T03:41:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534624+0000", 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4298e-05 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.534633+0000", 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5759999999999995e-06 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.543482+0000", 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088489710000000006 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.543521+0000", 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8902999999999997e-05 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 74/55 ECSubWrite(tid=30, reqid=osd.3.0:30, at_version=74'47, trim_to=0'0, roll_forward_to=73'39))", 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.536410+0000", 2024-09-16T03:41:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.696251447, 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089080830000000007, 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536410+0000", 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536410+0000", 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536410+0000", 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536430+0000", 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9772999999999999e-05 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536448+0000", 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8434e-05 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536455+0000", 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1069999999999999e-06 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.540745+0000", 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0042902449999999998 2024-09-16T03:41:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.540755+0000", 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3449999999999995e-06 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.545279+0000", 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045243419999999998 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.545318+0000", 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8510999999999998e-05 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.9s4 84/55 ECSubWrite(tid=174, reqid=osd.1.0:174, at_version=84'76, trim_to=0'0, roll_forward_to=82'74))", 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.479168+0000", 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7534933580000001, 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014893587999999999, 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479168+0000", 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479168+0000", 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479168+0000", 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1299999999999999e-07 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479192+0000", 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4284999999999999e-05 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479202+0000", 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5109999999999992e-06 2024-09-16T03:41:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.479211+0000", 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3640000000000005e-06 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480030+0000", 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00081856500000000005 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480041+0000", 2024-09-16T03:41:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1117999999999999e-05 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.494028+0000", 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013986801 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.494061+0000", 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3730999999999999e-05 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.0s2 84/31 ECSubWrite(tid=175, reqid=osd.1.0:175, at_version=84'80, trim_to=0'0, roll_forward_to=82'78))", 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.480016+0000", 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7526453440000003, 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010797957, 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480016+0000", 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480016+0000", 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480016+0000", 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7500000000000001e-07 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480039+0000", 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2920000000000001e-05 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480048+0000", 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1139999999999995e-06 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480056+0000", 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7360000000000001e-06 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480104+0000", 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8513999999999997e-05 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480113+0000", 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.02e-06 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.490776+0000", 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010662443000000001 2024-09-16T03:41:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.490814+0000", 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7834999999999999e-05 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.9s4 84/55 ECSubWrite(tid=176, reqid=osd.1.0:176, at_version=84'78, trim_to=0'0, roll_forward_to=82'74))", 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.480196+0000", 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7524650580000003, 2024-09-16T03:41:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024949954, 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480196+0000", 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480196+0000", 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480197+0000", 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0900000000000001e-07 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480222+0000", 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5853000000000001e-05 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480233+0000", 2024-09-16T03:41:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0862e-05 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.480240+0000", 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1049999999999997e-06 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.487747+0000", 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075061930000000004 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.487757+0000", 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0071000000000001e-05 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.505055+0000", 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017298153 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.505146+0000", 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1107999999999996e-05 2024-09-16T03:41:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.0s2 84/31 ECSubWrite(tid=177, reqid=osd.1.0:177, at_version=84'82, trim_to=0'0, roll_forward_to=82'78))", 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.482547+0000", 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7501143829999997, 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013538401, 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.482547+0000", 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.482547+0000", 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.482547+0000", 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.68e-07 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.482579+0000", 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1377999999999997e-05 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.482589+0000", 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9219999999999992e-06 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.482596+0000", 2024-09-16T03:41:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1879999999999996e-06 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.482670+0000", 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4453e-05 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.482680+0000", 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.482e-06 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.496051+0000", 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013371545 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.496085+0000", 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3764999999999999e-05 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 84/55 ECSubWrite(tid=50, reqid=osd.3.0:50, at_version=84'71, trim_to=0'0, roll_forward_to=83'69))", 2024-09-16T03:41:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.483403+0000", 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7492582910000003, 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018793977999999999, 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483403+0000", 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483403+0000", 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483403+0000", 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.84e-07 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483432+0000", 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8387000000000001e-05 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483443+0000", 2024-09-16T03:41:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1358e-05 2024-09-16T03:41:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483452+0000", 2024-09-16T03:41:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2240000000000001e-06 2024-09-16T03:41:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483521+0000", 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8592999999999999e-05 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483538+0000", 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7584999999999999e-05 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.502173+0000", 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018634973999999999 2024-09-16T03:41:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.502197+0000", 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3473e-05 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.7s1 84/55 ECSubWrite(tid=75, reqid=osd.9.0:75, at_version=84'58, trim_to=0'0, roll_forward_to=82'56))", 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.483568+0000", 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7490935429999999, 2024-09-16T03:41:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015146402, 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483568+0000", 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483568+0000", 2024-09-16T03:41:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483568+0000", 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483587+0000", 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8842e-05 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483597+0000", 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0407999999999999e-05 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483605+0000", 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6890000000000004e-06 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483677+0000", 2024-09-16T03:41:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1755999999999995e-05 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.483686+0000", 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8310000000000007e-06 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.498665+0000", 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014979653000000001 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.498714+0000", 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8525e-05 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.es3 84/55 ECSubWrite(tid=51, reqid=osd.3.0:51, at_version=84'73, trim_to=0'0, roll_forward_to=83'69))", 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.485235+0000", 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7474263099999998, 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019844878999999999, 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.485235+0000", 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.485235+0000", 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.485235+0000", 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0699999999999996e-07 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.485256+0000", 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0769999999999999e-05 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.485268+0000", 2024-09-16T03:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2148e-05 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.485276+0000", 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7190000000000001e-06 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.498665+0000", 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013388789999999999 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.498675+0000", 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.505038+0000", 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063629810000000002 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.505080+0000", 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1977999999999997e-05 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 84/31 ECSubWrite(tid=177, reqid=osd.6.0:177, at_version=84'156, trim_to=0'0, roll_forward_to=83'154))", 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.491430+0000", 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7412306529999997, 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013756766, 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.491430+0000", 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.491430+0000", 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.491431+0000", 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6300000000000005e-07 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.491451+0000", 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0259e-05 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.491461+0000", 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9359999999999994e-06 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.491473+0000", 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2214e-05 2024-09-16T03:41:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.494049+0000", 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025760610000000001 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.494055+0000", 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.505151+0000", 2024-09-16T03:41:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011095285999999999 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.505187+0000", 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6504000000000002e-05 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.1s1 84/31 ECSubWrite(tid=178, reqid=osd.6.0:178, at_version=84'158, trim_to=0'0, roll_forward_to=83'154))", 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:41:03.492129+0000", 2024-09-16T03:41:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.740532204, 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013248467, 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.492129+0000", 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.492129+0000", 2024-09-16T03:41:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.492129+0000", 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0100000000000005e-07 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.492148+0000", 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8397999999999999e-05 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.492157+0000", 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.492165+0000", 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.503e-06 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.501607+0000", 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0094421160000000004 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.501618+0000", 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1398e-05 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.505355+0000", 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037362580000000001 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:41:03.505377+0000", 2024-09-16T03:41:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2603999999999999e-05 2024-09-16T03:41:08.329 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.329 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.329 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.329 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.329 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.329 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:08.331 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:41:08.331 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:08.329+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:08.431 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:08.429+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:08.475 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:08.475 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:08.475 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:41:08.475 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:08.484 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:41:08.531 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:08.530+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:08.588 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T03:41:08.588 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:08.588 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:08.588 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T03:41:08.588 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T03:41:08.588 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:08.592 INFO:tasks.rados.rados.0.smithi121.stdout:350: done (1 left) 2024-09-16T03:41:08.592 INFO:tasks.rados.rados.0.smithi121.stdout:351: done (0 left) 2024-09-16T03:41:08.592 INFO:tasks.rados.rados.0.smithi121.stdout:352: snap_create 2024-09-16T03:41:08.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:41:08.632 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:08.631+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:08.708 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:08.708 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T03:41:08.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T03:41:08.708 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T03:41:08.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.709 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:229 3.5s0 3:a4bec39a:::smithi12138290-4:head [copy-from ver 9] snapc 4=[4,3] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:41:08.709 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:41:08.709 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 51.165116517999998, 2024-09-16T03:41:08.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016562443, 2024-09-16T03:41:08.709 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.709 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:08.709 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.709 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 229 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540576+0000", 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540579+0000", 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6910000000000002e-06 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540591+0000", 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2018000000000001e-05 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540631+0000", 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9938999999999998e-05 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540643+0000", 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1902000000000001e-05 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540731+0000", 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8585000000000007e-05 2024-09-16T03:41:08.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540736+0000", 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9919999999999998e-06 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542533+0000", 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017970950000000001 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.542727+0000", 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019304700000000001 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.553519+0000", 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010792568000000001 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554988+0000", 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014687490000000001 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557059+0000", 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00207092 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.557139+0000", 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9937000000000004e-05 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.715 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:230 3.5s0 3:a4bec39a:::smithi12138290-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 51.165005413000003, 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014084419000000001, 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 230 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540687+0000", 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540688+0000", 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:41:08.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540692+0000", 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.019e-06 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540724+0000", 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2237000000000003e-05 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540740+0000", 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6075000000000001e-05 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540783+0000", 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2249000000000002e-05 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.540786+0000", 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4869999999999998e-06 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543015+0000", 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022290859999999999 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.543063+0000", 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7496000000000001e-05 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554628+0000", 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011565203 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554697+0000", 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9381999999999994e-05 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:17.554772+0000", 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4535000000000004e-05 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:314 3.5s0 3:a4bec39a:::smithi12138290-4:head [stat,delete] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:41:08.721 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.214939420999997, 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010497464, 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 314 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490753+0000", 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490755+0000", 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.736e-06 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490767+0000", 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1931e-05 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490783+0000", 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6135000000000001e-05 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490791+0000", 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1620000000000008e-06 2024-09-16T03:41:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490849+0000", 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7581000000000003e-05 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.490856+0000", 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9299999999999997e-06 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.492657+0000", 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018011539999999999 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.492731+0000", 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4263000000000004e-05 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.493469+0000", 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073743400000000003 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.495358+0000", 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001888898 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.501195+0000", 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058370879999999998 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:28.501251+0000", 2024-09-16T03:41:08.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6152e-05 2024-09-16T03:41:08.728 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.728 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.728 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.728 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 53/31 ECSubWrite(tid=27, reqid=client.4615.0:354, at_version=53'8, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:41:08.728 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:41:08.728 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.200741892000003, 2024-09-16T03:41:08.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055158589999999997, 2024-09-16T03:41:08.729 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.729 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.729 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.729 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:41:08.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504951+0000", 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1e-07 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504976+0000", 2024-09-16T03:41:08.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5259999999999999e-05 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504984+0000", 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6850000000000001e-06 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.504993+0000", 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4299999999999995e-06 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505310+0000", 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031660299999999999 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.505342+0000", 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.15e-05 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510423+0000", 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050819000000000003 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:32.510467+0000", 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3271e-05 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:467 3.5s0 3:a45e0081:::smithi12138290-25:head [copy-from ver 16] snapc 11=[11,10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.222861677000001, 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014721097000000001, 2024-09-16T03:41:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 467 2024-09-16T03:41:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482831+0000", 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482832+0000", 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3e-06 2024-09-16T03:41:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482839+0000", 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3010000000000003e-06 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482849+0000", 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9499999999999996e-06 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482854+0000", 2024-09-16T03:41:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1410000000000004e-06 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482916+0000", 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2040999999999998e-05 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483028+0000", 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112674 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.491088+0000", 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080595969999999999 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.493079+0000", 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019907990000000001 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.497458+0000", 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004378804 2024-09-16T03:41:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.497552+0000", 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4489999999999998e-05 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:468 3.5s0 3:a45e0081:::smithi12138290-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.222809697999999, 2024-09-16T03:41:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010159837999999999, 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 468 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:41:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482883+0000", 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482884+0000", 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.16e-07 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482888+0000", 2024-09-16T03:41:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0099999999999997e-06 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482903+0000", 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5418999999999999e-05 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.482910+0000", 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6850000000000004e-06 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483315+0000", 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00040511699999999998 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.483369+0000", 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3884000000000001e-05 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492657+0000", 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092879590000000001 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.492897+0000", 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023993300000000001 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:38.493043+0000", 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014601499999999999 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.fs1 73/55 ECSubWrite(tid=174, reqid=osd.0.0:174, at_version=73'30, trim_to=0'0, roll_forward_to=56'28))", 2024-09-16T03:41:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.478077+0000", 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.227615341, 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060656310000000001, 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478077+0000", 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478077+0000", 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478078+0000", 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478111+0000", 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3151000000000002e-05 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478122+0000", 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1420000000000001e-05 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478132+0000", 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7280000000000007e-06 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478185+0000", 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2837999999999998e-05 2024-09-16T03:41:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.478194+0000", 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6959999999999994e-06 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484096+0000", 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059024389999999998 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484143+0000", 2024-09-16T03:41:08.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7079000000000001e-05 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.as6 73/55 ECSubWrite(tid=117, reqid=osd.12.0:114, at_version=0'0, trim_to=0'0, roll_forward_to=73'28))", 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.480476+0000", 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.22521708, 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063949760000000001, 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480476+0000", 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480476+0000", 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480476+0000", 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480486+0000", 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6309999999999998e-06 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480500+0000", 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3767999999999999e-05 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480508+0000", 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2530000000000004e-06 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480592+0000", 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3918999999999994e-05 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480601+0000", 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5130000000000002e-06 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.486838+0000", 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062362119999999997 2024-09-16T03:41:08.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.486871+0000", 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2981999999999997e-05 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 73/31 ECSubWrite(tid=118, reqid=osd.12.0:116, at_version=0'0, trim_to=0'0, roll_forward_to=73'54))", 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.480541+0000", 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.225152130000001, 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064373980000000004, 2024-09-16T03:41:08.753 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480541+0000", 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480541+0000", 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480541+0000", 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9900000000000002e-07 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480554+0000", 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3071000000000001e-05 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480563+0000", 2024-09-16T03:41:08.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9889999999999998e-06 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480612+0000", 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8878000000000002e-05 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480767+0000", 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015537400000000001 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.480776+0000", 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9560000000000003e-06 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.486953+0000", 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061768220000000002 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.486978+0000", 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5009000000000001e-05 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 73/31 ECSubWrite(tid=119, reqid=osd.12.0:119, at_version=73'56, trim_to=0'0, roll_forward_to=73'54))", 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.481500+0000", 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.224192359, 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079937859999999993, 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481500+0000", 2024-09-16T03:41:08.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481500+0000", 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481501+0000", 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3499999999999995e-07 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481520+0000", 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8533999999999999e-05 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481540+0000", 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0389000000000001e-05 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481548+0000", 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1259999999999998e-06 2024-09-16T03:41:08.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481613+0000", 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4640000000000005e-05 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.481621+0000", 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5690000000000005e-06 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489462+0000", 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078409450000000002 2024-09-16T03:41:08.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489494+0000", 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1847999999999997e-05 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 73/31 ECSubWrite(tid=91, reqid=osd.14.0:91, at_version=73'73, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.482669+0000", 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.223023852000001, 2024-09-16T03:41:08.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0068390079999999997, 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482669+0000", 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482669+0000", 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482669+0000", 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.54e-07 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482686+0000", 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.713e-05 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482694+0000", 2024-09-16T03:41:08.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3049999999999993e-06 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482700+0000", 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0629999999999996e-06 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482759+0000", 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9052999999999999e-05 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.482769+0000", 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0759e-05 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489458+0000", 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066889030000000004 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.489508+0000", 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9641000000000001e-05 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.6s3 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.483429+0000", 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.222263636000001, 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072320709999999996, 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483429+0000", 2024-09-16T03:41:08.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483429+0000", 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483430+0000", 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2799999999999996e-07 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483443+0000", 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3332e-05 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483451+0000", 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.977e-06 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.483456+0000", 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4510000000000002e-06 2024-09-16T03:41:08.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484487+0000", 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001031129 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484495+0000", 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5280000000000003e-06 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490628+0000", 2024-09-16T03:41:08.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061330730000000002 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490661+0000", 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2953000000000001e-05 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.3s1 73/31 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=73'31, trim_to=0'0, roll_forward_to=66'29))", 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:52.484560+0000", 2024-09-16T03:41:08.771 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.221132639, 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057403109999999997, 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484560+0000", 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484560+0000", 2024-09-16T03:41:08.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484560+0000", 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4900000000000002e-07 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484581+0000", 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0741999999999999e-05 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484592+0000", 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1087000000000001e-05 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484601+0000", 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6729999999999998e-06 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484654+0000", 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3606999999999997e-05 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.484660+0000", 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7559999999999996e-06 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490261+0000", 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056009620000000001 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:52.490300+0000", 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9235e-05 2024-09-16T03:41:08.775 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:685 3.5s0 3:a45e0081:::smithi12138290-25:head [read 0~1409024,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.529168+0000", 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.176525163999999, 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070249730000000003, 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.776 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 685 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529168+0000", 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529168+0000", 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529170+0000", 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8700000000000001e-06 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529180+0000", 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3500000000000003e-06 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529203+0000", 2024-09-16T03:41:08.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3410000000000001e-05 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529214+0000", 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0360000000000001e-05 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529367+0000", 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015317600000000001 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529468+0000", 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000101619 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536193+0000", 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067241879999999999 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:686 3.5s0 3:a45e0081:::smithi12138290-25:head [read 0~1409024,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:08.780 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.529236+0000", 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.176456297, 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090419060000000006, 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 686 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529236+0000", 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529236+0000", 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529237+0000", 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529241+0000", 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2429999999999999e-06 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529249+0000", 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6229999999999997e-06 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529255+0000", 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6439999999999997e-06 2024-09-16T03:41:08.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533107+0000", 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038518879999999999 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.533199+0000", 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2863999999999997e-05 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.538278+0000", 2024-09-16T03:41:08.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050790059999999996 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:687 3.5s0 3:a45e0081:::smithi12138290-25:head [read 0~1409024,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.529263+0000", 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.176429690000001, 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01182183, 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 687 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529263+0000", 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529263+0000", 2024-09-16T03:41:08.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529264+0000", 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3600000000000004e-07 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529267+0000", 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1190000000000001e-06 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529273+0000", 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4649999999999999e-06 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.529278+0000", 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6059999999999996e-06 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536430+0000", 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071525820000000002 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.536515+0000", 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4332999999999994e-05 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.541085+0000", 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045701889999999997 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDECSubOpWrite(3.ds4 74/31 ECSubWrite(tid=128, reqid=client.4615.0:680, at_version=74'63, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:53.537219+0000", 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.168474117000001, 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056221450000000003, 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537219+0000", 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537219+0000", 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537219+0000", 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-07 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537235+0000", 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5577e-05 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537243+0000", 2024-09-16T03:41:08.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8049999999999996e-06 2024-09-16T03:41:08.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.537251+0000", 2024-09-16T03:41:08.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.126e-06 2024-09-16T03:41:08.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.538933+0000", 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001682006 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_started", 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.538941+0000", 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8859999999999993e-06 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_committed", 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542806+0000", 2024-09-16T03:41:08.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038642699999999999 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:53.542841+0000", 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5105999999999999e-05 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:778 3.5s0 3:a4bec39a:::smithi12138290-4:1b [read 0~1343488,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:56.583639+0000", 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.122053551, 2024-09-16T03:41:08.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007324077, 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 778 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583639+0000", 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583639+0000", 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583641+0000", 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8619999999999999e-06 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583648+0000", 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8859999999999998e-06 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583665+0000", 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6962000000000001e-05 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583675+0000", 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0127e-05 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583740+0000", 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5353999999999998e-05 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583845+0000", 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104431 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.590963+0000", 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071184550000000001 2024-09-16T03:41:08.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:779 3.5s0 3:a4bec39a:::smithi12138290-4:1b [read 0~1343488,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:56.583697+0000", 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.121995800000001, 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079483090000000006, 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.799 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 779 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583697+0000", 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583697+0000", 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583698+0000", 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4499999999999997e-07 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583701+0000", 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7790000000000002e-06 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583709+0000", 2024-09-16T03:41:08.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2120000000000003e-06 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583715+0000", 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7579999999999998e-06 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.584246+0000", 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00053043999999999997 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.584344+0000", 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8442000000000004e-05 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.591645+0000", 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073011329999999996 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4615.0:780 3.5s0 3:a4bec39a:::smithi12138290-4:1b [read 0~1343488,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:08.803 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T03:40:56.583722+0000", 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.121970943999999, 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085727600000000004, 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "continuous": false, 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4615", 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 780 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T03:41:08.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583722+0000", 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583722+0000", 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583722+0000", 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-07 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583725+0000", 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9699999999999999e-06 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583743+0000", 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7632e-05 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.583748+0000", 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7570000000000004e-06 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.586247+0000", 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024994029999999999 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.586333+0000", 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6497999999999998e-05 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T03:40:56.592295+0000", 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005961142 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.807 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:08.808 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T03:41:08.808 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:08.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:41:08.810 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:08.807+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:08.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:41:08.910 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:08.909+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:08.922 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:08.923 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:08.923 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:41:08.923 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:08.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:41:08.973 INFO:teuthology.orchestra.run.smithi076.stderr:Exporting 3.cs2 info 3.cs2( v 82'155 (0'0,82'155] local-lis/les=31/32 n=19 ec=31/31 lis/c=31/31 les/c/f=32/32/0 sis=31) 2024-09-16T03:41:08.978 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:32badfe7:::smithi12138290-17:15# 2024-09-16T03:41:08.978 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:32badfe7:::smithi12138290-17:19# 2024-09-16T03:41:08.979 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:32badfe7:::smithi12138290-17:head# 2024-09-16T03:41:08.981 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:335b675a:::smithi12138290-38:e# 2024-09-16T03:41:08.983 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:335b675a:::smithi12138290-38:head# 2024-09-16T03:41:08.983 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:3432dbf3:::smithi12138290-44:11# 2024-09-16T03:41:08.984 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:3432dbf3:::smithi12138290-44:1b# 2024-09-16T03:41:08.984 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:3432dbf3:::smithi12138290-44:head# 2024-09-16T03:41:08.984 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:38c596e1:::smithi12138290-26:11# 2024-09-16T03:41:08.984 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:38c596e1:::smithi12138290-26:19# 2024-09-16T03:41:08.985 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:38c596e1:::smithi12138290-26:1b# 2024-09-16T03:41:08.985 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:38c596e1:::smithi12138290-26:head# 2024-09-16T03:41:08.986 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:39b8e90f:::smithi12138290-2:17# 2024-09-16T03:41:08.987 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:39b8e90f:::smithi12138290-2:1a# 2024-09-16T03:41:08.988 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:39b8e90f:::smithi12138290-2:head# 2024-09-16T03:41:08.991 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:3aac74e6:::smithi12138290-45:d# 2024-09-16T03:41:08.991 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:41:08.992 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:41:08.993 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:3aac74e6:::smithi12138290-45:head# 2024-09-16T03:41:08.993 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:3e925a6e:::smithi12138290-43:1b# 2024-09-16T03:41:08.993 INFO:teuthology.orchestra.run.smithi076.stderr:Read 2#3:3e925a6e:::smithi12138290-43:head# 2024-09-16T03:41:08.993 INFO:teuthology.orchestra.run.smithi076.stderr:Export successful 2024-09-16T03:41:09.009 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:41:09.011 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:09.010+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:09.026 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:41:09.026 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:41:09.035 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:09.036 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:09.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:41:09.045 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:41:09.045 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:41:09.060 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:41:09.060 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:41:09.077 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:41:09.077 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:41:09.093 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:41:09.094 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:41:09.110 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:41:09.111 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:41:09.111 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:09.110+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:09.126 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:41:09.126 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:41:09.143 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:41:09.143 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:41:09.160 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:41:09.160 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:41:09.177 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:41:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:41:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-16T03:41:09.189 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:09.189 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:09.194 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:41:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:41:09.198 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:41:09.211 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:41:09.211 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:41:09.212 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:09.211+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:09.228 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:41:09.228 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:41:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:41:09.232 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:41:09.302 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:09.302 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:09.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:41:09.313 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:09.311+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:09.413 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:09.413 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:09.414 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:09.412+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:09.422 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:41:09.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-12 --journal-path /var/lib/ceph/osd/ceph-12/journal --force --op remove --pgid 3.cs2 2024-09-16T03:41:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:09.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:41:09.549 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:09.548+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:09.607 INFO:tasks.rados.rados.0.smithi121.stdout:352: done (0 left) 2024-09-16T03:41:09.607 INFO:tasks.rados.rados.0.smithi121.stdout:353: snap_remove snap 24 2024-09-16T03:41:09.615 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:41:09.614+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:41:09.637 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:09.637 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:09.646 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:41:09.730 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:09.729+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:09.791 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:09.791 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:09.800 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:41:09.817 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:09.816+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:09.918 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:09.917+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:09.944 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:09.944 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:09.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:41:10.019 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:10.018+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:10.069 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:10.069 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:10.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:41:10.120 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:41:10.118+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:41:10.182 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:10.182 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:10.191 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:41:10.220 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:10.219+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:10.294 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:10.294 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:10.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:41:10.321 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:41:10.319+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:41:10.406 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:10.407 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:10.415 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:41:10.421 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:10.420+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:10.517 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:10.517 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:10.522 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:10.520+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:10.526 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:41:10.606 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:41:10.614 INFO:tasks.rados.rados.0.smithi121.stdout:353: done (0 left) 2024-09-16T03:41:10.614 INFO:tasks.rados.rados.0.smithi121.stdout:354: read oid 15 snap -1 2024-09-16T03:41:10.614 INFO:tasks.rados.rados.0.smithi121.stdout:354: expect (ObjNum 56 snap 0 seq_num 56) 2024-09-16T03:41:10.614 INFO:tasks.rados.rados.0.smithi121.stdout:355: rmattr oid 34 current snap is 34 2024-09-16T03:41:10.614 INFO:tasks.rados.rados.0.smithi121.stdout:356: rmattr oid 22 current snap is 34 2024-09-16T03:41:10.614 INFO:tasks.rados.rados.0.smithi121.stdout:356: done (2 left) 2024-09-16T03:41:10.614 INFO:tasks.rados.rados.0.smithi121.stdout:357: read oid 46 snap -1 2024-09-16T03:41:10.614 INFO:tasks.rados.rados.0.smithi121.stdout:357: expect (ObjNum 87 snap 12 seq_num 87) 2024-09-16T03:41:10.615 INFO:tasks.rados.rados.0.smithi121.stdout:358: delete oid 2 current snap is 34 2024-09-16T03:41:10.622 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 34 v 65 (ObjNum 153 snap 31 seq_num 153) dirty exists 2024-09-16T03:41:10.633 INFO:tasks.rados.rados.0.smithi121.stdout:354: expect (ObjNum 56 snap 0 seq_num 56) 2024-09-16T03:41:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:10.650 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:10.649+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:10.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:41:10.672 INFO:tasks.rados.rados.0.smithi121.stdout:357: expect (ObjNum 87 snap 12 seq_num 87) 2024-09-16T03:41:10.723 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:10.722+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:10.739 INFO:tasks.rados.rados.0.smithi121.stdout:354: done (3 left) 2024-09-16T03:41:10.739 INFO:tasks.rados.rados.0.smithi121.stdout:355: done (2 left) 2024-09-16T03:41:10.740 INFO:tasks.rados.rados.0.smithi121.stdout:357: done (1 left) 2024-09-16T03:41:10.740 INFO:tasks.rados.rados.0.smithi121.stdout:358: done (0 left) 2024-09-16T03:41:10.740 INFO:tasks.rados.rados.0.smithi121.stdout:359: rmattr oid 4 current snap is 34 2024-09-16T03:41:10.740 INFO:tasks.rados.rados.0.smithi121.stdout:359: done (0 left) 2024-09-16T03:41:10.740 INFO:tasks.rados.rados.0.smithi121.stdout:360: snap_create 2024-09-16T03:41:10.757 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:10.757 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:10.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:41:10.824 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:10.822+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:10.869 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:10.869 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:10.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:41:10.925 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:10.923+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:11.022 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:11.022 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:11.025 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:11.024+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:11.030 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:41:11.127 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:41:11.125+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:41:11.132 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:11.132 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:11.141 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:41:11.227 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:11.225+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:11.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:41:11.242 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:11.243 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:11.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:41:11.329 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:11.327+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:11.354 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:41:11.355 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:41:11.372 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:41:11.372 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:41:11.389 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:41:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:41:11.395 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:11.395 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:11.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:41:11.406 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:41:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:41:11.422 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:41:11.422 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:41:11.429 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:11.428+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:11.439 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:41:11.439 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:41:11.441 INFO:teuthology.orchestra.run.smithi076.stdout: marking collection for removal 2024-09-16T03:41:11.441 INFO:teuthology.orchestra.run.smithi076.stdout:setting '_remove' omap key 2024-09-16T03:41:11.441 INFO:teuthology.orchestra.run.smithi076.stdout:finish_remove_pgs 3.cs2_head removing 3.cs2 2024-09-16T03:41:11.448 INFO:teuthology.orchestra.run.smithi076.stdout:Remove successful 2024-09-16T03:41:11.459 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:41:11.459 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:41:11.475 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:41:11.475 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:41:11.476 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T03:41:11.479 INFO:tasks.rados.rados.0.smithi121.stdout:360: done (0 left) 2024-09-16T03:41:11.479 INFO:tasks.rados.rados.0.smithi121.stdout:361: copy_from oid 5 from oid 35 current snap is 35 2024-09-16T03:41:11.480 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 33 current snap is 35 2024-09-16T03:41:11.480 INFO:tasks.rados.rados.0.smithi121.stdout:362: seq_num 156 ranges {2605056=229376} 2024-09-16T03:41:11.493 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:41:11.493 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:41:11.495 INFO:tasks.rados.rados.0.smithi121.stdout:362: writing smithi12138290-33 from 2605056 to 2834432 tid 1 2024-09-16T03:41:11.495 INFO:tasks.rados.rados.0.smithi121.stdout:363: read oid 1 snap 15 2024-09-16T03:41:11.495 INFO:tasks.rados.rados.0.smithi121.stdout:363: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-16T03:41:11.495 INFO:tasks.rados.rados.0.smithi121.stdout:364: copy_from oid 30 from oid 37 current snap is 35 2024-09-16T03:41:11.495 INFO:tasks.rados.rados.0.smithi121.stdout:361: finishing copy_from to smithi12138290-5 2024-09-16T03:41:11.495 INFO:tasks.rados.rados.0.smithi121.stdout:361: got expected ENOENT (src dne) 2024-09-16T03:41:11.495 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 32 current snap is 35 2024-09-16T03:41:11.496 INFO:tasks.rados.rados.0.smithi121.stdout:361: finishing copy_from racing read to smithi12138290-5 2024-09-16T03:41:11.496 INFO:tasks.rados.rados.0.smithi121.stdout:365: seq_num 157 ranges {4259840=245760} 2024-09-16T03:41:11.512 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:41:11.512 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:41:11.521 INFO:tasks.rados.rados.0.smithi121.stdout:365: writing smithi12138290-32 from 4259840 to 4505600 tid 1 2024-09-16T03:41:11.521 INFO:tasks.rados.rados.0.smithi121.stdout:361: done (4 left) 2024-09-16T03:41:11.521 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 47 current snap is 35 2024-09-16T03:41:11.521 INFO:tasks.rados.rados.0.smithi121.stdout:366: seq_num 158 ranges {0=262144} 2024-09-16T03:41:11.522 INFO:tasks.rados.rados.0.smithi121.stdout:366: writing smithi12138290-47 from 0 to 262144 tid 1 2024-09-16T03:41:11.522 INFO:tasks.rados.rados.0.smithi121.stdout:367: snap_create 2024-09-16T03:41:11.522 INFO:tasks.rados.rados.0.smithi121.stdout:363: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-16T03:41:11.527 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:11.527 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:11.527 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:41:11.527 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:11.530 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:41:11.529+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:41:11.531 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:41:11.531 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:41:11.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:41:11.539 INFO:tasks.rados.rados.0.smithi121.stdout:362: finishing write tid 1 to smithi12138290-33 2024-09-16T03:41:11.539 INFO:tasks.rados.rados.0.smithi121.stdout:362: finishing write tid 2 to smithi12138290-33 2024-09-16T03:41:11.539 INFO:tasks.rados.rados.0.smithi121.stdout:362: finishing write tid 3 to smithi12138290-33 2024-09-16T03:41:11.539 INFO:tasks.rados.rados.0.smithi121.stdout:362: oid 33 updating version 0 to 37 2024-09-16T03:41:11.539 INFO:tasks.rados.rados.0.smithi121.stdout:362: oid 33 updating version 37 to 38 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 33 v 38 (ObjNum 156 snap 35 seq_num 156) dirty exists 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:362: left oid 33 (ObjNum 156 snap 35 seq_num 156) 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:364: finishing copy_from racing read to smithi12138290-30 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:364: finishing copy_from to smithi12138290-30 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 30 v 34 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:365: finishing write tid 1 to smithi12138290-32 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:365: finishing write tid 2 to smithi12138290-32 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:366: finishing write tid 1 to smithi12138290-47 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:365: finishing write tid 3 to smithi12138290-32 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:365: oid 32 updating version 0 to 182 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:365: oid 32 updating version 182 to 183 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 32 v 183 (ObjNum 157 snap 35 seq_num 157) dirty exists 2024-09-16T03:41:11.540 INFO:tasks.rados.rados.0.smithi121.stdout:365: left oid 32 (ObjNum 157 snap 35 seq_num 157) 2024-09-16T03:41:11.541 INFO:tasks.rados.rados.0.smithi121.stdout:366: finishing write tid 2 to smithi12138290-47 2024-09-16T03:41:11.541 INFO:tasks.rados.rados.0.smithi121.stdout:366: finishing write tid 3 to smithi12138290-47 2024-09-16T03:41:11.541 INFO:tasks.rados.rados.0.smithi121.stdout:366: oid 47 updating version 0 to 90 2024-09-16T03:41:11.541 INFO:tasks.rados.rados.0.smithi121.stdout:366: oid 47 updating version 90 to 91 2024-09-16T03:41:11.541 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 47 v 91 (ObjNum 158 snap 35 seq_num 158) dirty exists 2024-09-16T03:41:11.541 INFO:tasks.rados.rados.0.smithi121.stdout:366: left oid 47 (ObjNum 158 snap 35 seq_num 158) 2024-09-16T03:41:11.547 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:41:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:41:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-16T03:41:11.564 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:41:11.564 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:41:11.580 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:41:11.580 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:41:11.597 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:41:11.597 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:41:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:41:11.600 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:41:11.631 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:41:11.629+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:41:11.643 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:11.644 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:11.644 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:41:11.644 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:41:11.644 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:11.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:41:11.737 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:11.735+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=109, reqid=osd.6.0:109, at_version=73'87, trim_to=0'0, roll_forward_to=72'85))", 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.477672+0000", 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.317367385000001, 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056828829999999997, 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.798 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477672+0000", 2024-09-16T03:41:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477672+0000", 2024-09-16T03:41:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477672+0000", 2024-09-16T03:41:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4400000000000001e-07 2024-09-16T03:41:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477692+0000", 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9857999999999999e-05 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477703+0000", 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1453e-05 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477712+0000", 2024-09-16T03:41:11.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5390000000000007e-06 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477795+0000", 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2634999999999997e-05 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477805+0000", 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0135e-05 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483322+0000", 2024-09-16T03:41:11.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055167840000000003 2024-09-16T03:41:11.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483355+0000", 2024-09-16T03:41:11.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3235000000000003e-05 2024-09-16T03:41:11.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.803 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.804 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 73/31 ECSubWrite(tid=83, reqid=osd.1.0:83, at_version=73'38, trim_to=0'0, roll_forward_to=72'36))", 2024-09-16T03:41:11.804 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:11.804 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.317176696000001, 2024-09-16T03:41:11.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059263110000000001, 2024-09-16T03:41:11.804 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.804 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.804 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.804 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:11.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.17e-07 2024-09-16T03:41:11.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477878+0000", 2024-09-16T03:41:11.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4824000000000001e-05 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477887+0000", 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6509999999999996e-06 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477895+0000", 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.401e-06 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477945+0000", 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0555e-05 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477953+0000", 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.875e-06 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483761+0000", 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058074750000000003 2024-09-16T03:41:11.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483789+0000", 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8313000000000001e-05 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 73/31 ECSubWrite(tid=85, reqid=osd.1.0:85, at_version=73'40, trim_to=0'0, roll_forward_to=72'36))", 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.313879970999999, 2024-09-16T03:41:11.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071150040000000003, 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:41:11.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481160+0000", 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481177+0000", 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7368999999999999e-05 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481186+0000", 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1940000000000004e-06 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481194+0000", 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4209999999999998e-06 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481262+0000", 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8312000000000006e-05 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481272+0000", 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3519999999999996e-06 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488236+0000", 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069643860000000004 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488274+0000", 2024-09-16T03:41:11.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8390999999999999e-05 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=111, reqid=osd.6.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.303424548999999, 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006602218, 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491625+0000", 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7249999999999992e-06 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491634+0000", 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3370000000000006e-06 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491642+0000", 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0770000000000002e-06 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491764+0000", 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012253500000000001 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491772+0000", 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1499999999999999e-06 2024-09-16T03:41:11.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498186+0000", 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064136640000000003 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498217+0000", 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1080000000000001e-05 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 73/55 ECSubWrite(tid=98, reqid=osd.10.0:98, at_version=73'24, trim_to=0'0, roll_forward_to=56'20))", 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.301558601, 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065373489999999996, 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:11.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.61e-07 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493506+0000", 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4556999999999999e-05 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493520+0000", 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3804e-05 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493528+0000", 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9340000000000006e-06 2024-09-16T03:41:11.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493606+0000", 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8144000000000003e-05 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493617+0000", 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1368e-05 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499976+0000", 2024-09-16T03:41:11.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006358463 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500018+0000", 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2317999999999997e-05 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=112, reqid=osd.6.0:112, at_version=73'91, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:11.823 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.301234604000001, 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062537729999999998, 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5600000000000005e-07 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493823+0000", 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7804000000000001e-05 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493835+0000", 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2197e-05 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493845+0000", 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6409999999999997e-06 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493899+0000", 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4418e-05 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493909+0000", 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5670000000000001e-06 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500026+0000", 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006117468 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500058+0000", 2024-09-16T03:41:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2021999999999999e-05 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=113, reqid=osd.6.0:113, at_version=73'93, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.300951716, 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083691979999999996, 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.828 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0399999999999995e-07 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494110+0000", 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1750999999999999e-05 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494122+0000", 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1532000000000001e-05 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494129+0000", 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7300000000000005e-06 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495262+0000", 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011327679999999999 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495271+0000", 2024-09-16T03:41:11.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7739999999999993e-06 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502425+0000", 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071538160000000003 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502457+0000", 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2122999999999997e-05 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.832 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:682 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.267459148, 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095978390000000004, 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 682 2024-09-16T03:41:11.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527582+0000", 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7910000000000001e-06 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527590+0000", 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5320000000000001e-06 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527608+0000", 2024-09-16T03:41:11.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8496999999999999e-05 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527622+0000", 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3583e-05 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527699+0000", 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7799000000000005e-05 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527707+0000", 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.530874+0000", 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031660529999999998 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.530988+0000", 2024-09-16T03:41:11.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114376 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.531781+0000", 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079327 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536093+0000", 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043113819999999999 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.537011+0000", 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091858900000000004 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.537178+0000", 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016688799999999999 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:683 3.es0 3:71ce6426:::smithi12138290-50:head [rollback 18] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.267384435, 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015112265, 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 683 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:11.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:11.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:11.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:41:11.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527659+0000", 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4510000000000001e-06 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527667+0000", 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527673+0000", 2024-09-16T03:41:11.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0100000000000001e-06 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527773+0000", 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100029 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527780+0000", 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3239999999999999e-06 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.532223+0000", 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044434870000000003 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.532315+0000", 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2255000000000004e-05 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533195+0000", 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087904200000000002 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536094+0000", 2024-09-16T03:41:11.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028993840000000001 2024-09-16T03:41:11.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:11.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542627+0000", 2024-09-16T03:41:11.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006533021 2024-09-16T03:41:11.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542767+0000", 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014024 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:684 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.267355397999999, 2024-09-16T03:41:11.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015421104, 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 684 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000005e-07 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527687+0000", 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4609999999999998e-06 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527696+0000", 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7600000000000008e-06 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527701+0000", 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3789999999999997e-06 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527803+0000", 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010234900000000001 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527809+0000", 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2870000000000004e-06 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533474+0000", 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056649090000000001 2024-09-16T03:41:11.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533571+0000", 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7552000000000004e-05 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.534004+0000", 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00043247700000000001 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536095+0000", 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020918460000000001 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542953+0000", 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068572249999999998 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.543105+0000", 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015232000000000001 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.855 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=138, reqid=osd.6.0:138, at_version=78'117, trim_to=0'0, roll_forward_to=77'113))", 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.247779639999999, 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064033809999999997, 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.856 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547261+0000", 2024-09-16T03:41:11.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.032e-06 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547287+0000", 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5905e-05 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547301+0000", 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3905e-05 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547310+0000", 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4590000000000004e-06 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547537+0000", 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022751299999999999 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547553+0000", 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5113000000000001e-05 2024-09-16T03:41:11.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553619+0000", 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060667450000000001 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553663+0000", 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3708999999999998e-05 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 78/55 ECSubWrite(tid=122, reqid=osd.10.0:122, at_version=78'32, trim_to=0'0, roll_forward_to=76'30))", 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.24500252, 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006109504, 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9100000000000004e-07 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550065+0000", 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7671999999999999e-05 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550075+0000", 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5540000000000003e-06 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550083+0000", 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8119999999999997e-06 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550154+0000", 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0167999999999996e-05 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550163+0000", 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7850000000000003e-06 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556099+0000", 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059356310000000002 2024-09-16T03:41:11.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556146+0000", 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7391000000000001e-05 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=139, reqid=osd.6.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.240937191, 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078423550000000005, 2024-09-16T03:41:11.865 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554103+0000", 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7700000000000004e-07 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554111+0000", 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6119999999999999e-06 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554122+0000", 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0587999999999999e-05 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554129+0000", 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554193+0000", 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3522000000000006e-05 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554204+0000", 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.083e-05 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561904+0000", 2024-09-16T03:41:11.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077002540000000001 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561944+0000", 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0538999999999999e-05 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.as3 78/55 ECSubWrite(tid=161, reqid=osd.12.0:161, at_version=78'43, trim_to=0'0, roll_forward_to=77'41))", 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:41:11.870 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.240216972000001, 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067129479999999998, 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:41:11.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554823+0000", 2024-09-16T03:41:11.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:41:11.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554849+0000", 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6461999999999998e-05 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554862+0000", 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2918000000000001e-05 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554872+0000", 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.005e-05 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554971+0000", 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8665999999999994e-05 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554981+0000", 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0161e-05 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561484+0000", 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006503404 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561535+0000", 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0934999999999999e-05 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.875 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 78/31 ECSubWrite(tid=133, reqid=osd.1.0:133, at_version=78'56, trim_to=0'0, roll_forward_to=76'52))", 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.239713610000001, 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006360844, 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:41:11.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555327+0000", 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1299999999999998e-07 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555347+0000", 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0389000000000001e-05 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555359+0000", 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2371999999999999e-05 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555368+0000", 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0480000000000004e-06 2024-09-16T03:41:11.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555454+0000", 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5662999999999996e-05 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555464+0000", 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0018e-05 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561663+0000", 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061989549999999999 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561687+0000", 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3485999999999998e-05 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.as3 78/55 ECSubWrite(tid=163, reqid=osd.12.0:163, at_version=78'45, trim_to=0'0, roll_forward_to=77'41))", 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:41:11.880 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.238833678000001, 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078281010000000005, 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556207+0000", 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0809999999999999e-06 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556231+0000", 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.406e-05 2024-09-16T03:41:11.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556244+0000", 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3696999999999999e-05 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556258+0000", 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3342e-05 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556856+0000", 2024-09-16T03:41:11.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059772700000000005 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556865+0000", 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.166e-06 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564009+0000", 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007144358 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564034+0000", 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4669999999999999e-05 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=140, reqid=osd.6.0:140, at_version=78'119, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.237424452999999, 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064168259999999996, 2024-09-16T03:41:11.885 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557616+0000", 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9899999999999999e-07 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557640+0000", 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4034000000000001e-05 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557651+0000", 2024-09-16T03:41:11.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1382999999999999e-05 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557660+0000", 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557721+0000", 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1484999999999998e-05 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557731+0000", 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0051000000000001e-05 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.563993+0000", 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062610649999999997 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564032+0000", 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9193999999999998e-05 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:879 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.480957+0000", 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3140823830000006, 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091711669999999992, 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:11.890 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 879 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480957+0000", 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480957+0000", 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480959+0000", 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2259999999999999e-06 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480967+0000", 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.401e-06 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480983+0000", 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6704000000000001e-05 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480993+0000", 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0013000000000001e-05 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481088+0000", 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4381000000000003e-05 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481098+0000", 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8640000000000008e-06 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.483602+0000", 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025048539999999999 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.483734+0000", 2024-09-16T03:41:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000131617 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.484915+0000", 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001180918 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.489925+0000", 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050105779999999999 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490013+0000", 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7682000000000005e-05 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490128+0000", 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114929 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.896 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:880 3.es0 3:71ce6426:::smithi12138290-50:head [rollback 8] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3140195569999999, 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010599224000000001, 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 880 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:11.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2799999999999995e-07 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481024+0000", 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481036+0000", 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2088e-05 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481044+0000", 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5129999999999999e-06 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481172+0000", 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012779299999999999 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481177+0000", 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-09-16T03:41:11.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.485371+0000", 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041941410000000002 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.485473+0000", 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102734 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.486086+0000", 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061263300000000001 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490135+0000", 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040486020000000001 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491526+0000", 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001391155 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491619+0000", 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3103000000000006e-05 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:881 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3139555339999998, 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010772182999999999, 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:11.903 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 881 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9500000000000003e-07 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481089+0000", 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-09-16T03:41:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481096+0000", 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5549999999999997e-06 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481101+0000", 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4800000000000003e-06 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481212+0000", 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111263 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481220+0000", 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4069999999999993e-06 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.486523+0000", 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053031099999999998 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.486636+0000", 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112339 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.487220+0000", 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058448099999999998 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490409+0000", 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031889729999999999 2024-09-16T03:41:11.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491777+0000", 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001367458 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491856+0000", 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9231000000000001e-05 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:11.909 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:11.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:41:11.912 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:41:11.909+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:41:11.956 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-12 --journal-path /var/lib/ceph/osd/ceph-12/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.cs2.12 2024-09-16T03:41:12.012 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:41:12.011+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:41:12.056 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:12.056 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:12.056 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:41:12.056 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:12.064 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:41:12.113 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:12.111+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:12.166 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:12.167 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:12.167 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:41:12.167 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:41:12.167 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:12.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:41:12.220 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:12.219+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:12.280 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:41:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:41:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:41:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 42/31 ECSubWrite(tid=32, reqid=osd.14.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.713584183999998, 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075866950000000001, 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563725+0000", 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563726+0000", 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.09e-07 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563741+0000", 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5123000000000001e-05 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563761+0000", 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9335999999999999e-05 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563770+0000", 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3770000000000002e-06 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563840+0000", 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9725000000000001e-05 2024-09-16T03:41:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.563851+0000", 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1107000000000001e-05 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571277+0000", 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074264830000000002 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:21.571312+0000", 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4635000000000003e-05 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/31 ECSubWrite(tid=30, reqid=client.4615.0:301, at_version=49'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 43.790195115000003, 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083935009999999994, 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487114+0000", 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487115+0000", 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.36e-07 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487566+0000", 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045117600000000002 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487581+0000", 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5092000000000001e-05 2024-09-16T03:41:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487591+0000", 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.058e-05 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487677+0000", 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5635000000000003e-05 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.487685+0000", 2024-09-16T03:41:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2830000000000001e-06 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495470+0000", 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077842739999999999 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.495508+0000", 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8225000000000001e-05 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 49/31 ECSubWrite(tid=35, reqid=client.4615.0:306, at_version=49'23, trim_to=0'0, roll_forward_to=42'20))", 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 43.786992726999998, 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065860449999999996, 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:41:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490317+0000", 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490318+0000", 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5600000000000004e-07 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490342+0000", 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4006e-05 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490356+0000", 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4449e-05 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.490366+0000", 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8689999999999999e-06 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494506+0000", 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041400029999999997 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.494516+0000", 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0229e-05 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496868+0000", 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023520899999999998 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:28.496903+0000", 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4443000000000002e-05 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 73/31 ECSubWrite(tid=56, reqid=osd.8.0:56, at_version=73'42, trim_to=0'0, roll_forward_to=63'40))", 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.479531+0000", 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.797778595, 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083957719999999993, 2024-09-16T03:41:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479531+0000", 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479531+0000", 2024-09-16T03:41:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479531+0000", 2024-09-16T03:41:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2000000000000001e-07 2024-09-16T03:41:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479556+0000", 2024-09-16T03:41:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4822000000000001e-05 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479566+0000", 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7650000000000005e-06 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.479574+0000", 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7789999999999996e-06 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480246+0000", 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00067257600000000003 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480254+0000", 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3520000000000007e-06 2024-09-16T03:41:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.487884+0000", 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076291450000000004 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.487927+0000", 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3112999999999999e-05 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 73/31 ECSubWrite(tid=57, reqid=osd.8.0:57, at_version=73'44, trim_to=0'0, roll_forward_to=63'40))", 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.480913+0000", 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.796396221999998, 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087512579999999996, 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480913+0000", 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480913+0000", 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480914+0000", 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0699999999999996e-07 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480938+0000", 2024-09-16T03:41:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3949000000000001e-05 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480949+0000", 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0984e-05 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480957+0000", 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481866+0000", 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090959899999999998 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481875+0000", 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9709999999999993e-06 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.489628+0000", 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077526219999999998 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.489665+0000", 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6786000000000003e-05 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/31 ECSubWrite(tid=110, reqid=osd.6.0:110, at_version=73'89, trim_to=0'0, roll_forward_to=72'85))", 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.481009+0000", 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.796300413000001, 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0079369019999999992, 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481009+0000", 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481009+0000", 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481010+0000", 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8100000000000001e-07 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481030+0000", 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9636999999999999e-05 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481041+0000", 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1610999999999999e-05 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481049+0000", 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1429999999999998e-06 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481115+0000", 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5438999999999995e-05 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481125+0000", 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488908+0000", 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007783023 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488946+0000", 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8247000000000002e-05 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=91, reqid=osd.14.0:91, at_version=73'73, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.484293+0000", 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.79301628, 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012749313, 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484293+0000", 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484293+0000", 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484302+0000", 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3580000000000003e-06 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484318+0000", 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6327000000000001e-05 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484329+0000", 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0912e-05 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484336+0000", 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6320000000000008e-06 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484387+0000", 2024-09-16T03:41:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0361000000000001e-05 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484396+0000", 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5279999999999992e-06 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497001+0000", 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012604529999999999 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497043+0000", 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1665000000000002e-05 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.792957661999999, 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01525908, 2024-09-16T03:41:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.22e-07 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484387+0000", 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4653999999999997e-05 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484394+0000", 2024-09-16T03:41:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9550000000000003e-06 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484400+0000", 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7570000000000001e-06 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488364+0000", 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039642260000000004 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488373+0000", 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5880000000000004e-06 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499570+0000", 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011197087999999999 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499611+0000", 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.049e-05 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 73/55 ECSubWrite(tid=177, reqid=osd.0.0:174, at_version=0'0, trim_to=0'0, roll_forward_to=73'30))", 2024-09-16T03:41:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.488711+0000", 2024-09-16T03:41:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.788598818000001, 2024-09-16T03:41:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010894513, 2024-09-16T03:41:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488711+0000", 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488711+0000", 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488712+0000", 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6099999999999999e-07 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488726+0000", 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4591e-05 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488818+0000", 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1811999999999994e-05 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488829+0000", 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1114999999999999e-05 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488913+0000", 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3481999999999998e-05 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488922+0000", 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4069999999999999e-06 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499556+0000", 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010634403000000001 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499605+0000", 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8841999999999998e-05 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/31 ECSubWrite(tid=111, reqid=osd.6.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.490976+0000", 2024-09-16T03:41:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.786333674000002, 2024-09-16T03:41:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089424069999999994, 2024-09-16T03:41:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490976+0000", 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490976+0000", 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490977+0000", 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9199999999999999e-07 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490988+0000", 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1206000000000001e-05 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491004+0000", 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6351999999999999e-05 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491013+0000", 2024-09-16T03:41:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3489999999999992e-06 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492712+0000", 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016998219999999999 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492722+0000", 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3109999999999995e-06 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499886+0000", 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071638120000000003 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499918+0000", 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2663000000000002e-05 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 73/31 ECSubWrite(tid=58, reqid=osd.8.0:57, at_version=0'0, trim_to=0'0, roll_forward_to=73'44))", 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.492152+0000", 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.785157036000001, 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074894860000000001, 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492152+0000", 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492152+0000", 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492153+0000", 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5899999999999995e-07 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492165+0000", 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1952e-05 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492178+0000", 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2802e-05 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492187+0000", 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5459999999999992e-06 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492245+0000", 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8372000000000003e-05 2024-09-16T03:41:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492256+0000", 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0759e-05 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499611+0000", 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073555089999999997 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499642+0000", 2024-09-16T03:41:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0686999999999999e-05 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/31 ECSubWrite(tid=112, reqid=osd.6.0:112, at_version=73'91, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493729+0000", 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.783580812, 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085756970000000002, 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493729+0000", 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493729+0000", 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493729+0000", 2024-09-16T03:41:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4600000000000004e-07 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493757+0000", 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7419999999999998e-05 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493770+0000", 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3534e-05 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493779+0000", 2024-09-16T03:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3739999999999998e-06 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493870+0000", 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1544000000000003e-05 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493883+0000", 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2805e-05 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502272+0000", 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083892529999999993 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502304+0000", 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2020999999999997e-05 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/31 ECSubWrite(tid=113, reqid=osd.6.0:113, at_version=73'93, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.494706+0000", 2024-09-16T03:41:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.782603229999999, 2024-09-16T03:41:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013014379, 2024-09-16T03:41:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494706+0000", 2024-09-16T03:41:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494706+0000", 2024-09-16T03:41:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494707+0000", 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0500000000000003e-07 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494728+0000", 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1047999999999999e-05 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494742+0000", 2024-09-16T03:41:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3672000000000001e-05 2024-09-16T03:41:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494758+0000", 2024-09-16T03:41:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6184999999999998e-05 2024-09-16T03:41:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497457+0000", 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002698729 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497466+0000", 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.507681+0000", 2024-09-16T03:41:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010215284999999999 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.507721+0000", 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9715000000000002e-05 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=93, reqid=osd.14.0:92, at_version=0'0, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.500094+0000", 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.777215954999999, 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080041739999999993, 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500094+0000", 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500094+0000", 2024-09-16T03:41:12.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500095+0000", 2024-09-16T03:41:12.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.031e-06 2024-09-16T03:41:12.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500107+0000", 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2281e-05 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500118+0000", 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1294999999999999e-05 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500127+0000", 2024-09-16T03:41:12.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5350000000000004e-06 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500189+0000", 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2278000000000002e-05 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500198+0000", 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1009999999999998e-06 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508066+0000", 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078676249999999996 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508098+0000", 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2027999999999999e-05 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=94, reqid=osd.14.0:94, at_version=73'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.501070+0000", 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.776239485000001, 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007047439, 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501070+0000", 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501070+0000", 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501071+0000", 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501097+0000", 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6899000000000001e-05 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501107+0000", 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6520000000000001e-06 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501114+0000", 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9040000000000003e-06 2024-09-16T03:41:12.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501169+0000", 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5384999999999999e-05 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501179+0000", 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2189999999999993e-06 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508101+0000", 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069222160000000001 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508117+0000", 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6623999999999999e-05 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=95, reqid=osd.14.0:95, at_version=73'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.775245934000001, 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086042050000000002, 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0200000000000001e-07 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502084+0000", 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9769e-05 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502098+0000", 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3365e-05 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502105+0000", 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7430000000000002e-06 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502860+0000", 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075463899999999998 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502869+0000", 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8619999999999992e-06 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510637+0000", 2024-09-16T03:41:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077679530000000002 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510668+0000", 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1072000000000003e-05 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 78/31 ECSubWrite(tid=119, reqid=osd.14.0:119, at_version=78'103, trim_to=0'0, roll_forward_to=77'101))", 2024-09-16T03:41:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.552648+0000", 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.724661344999999, 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0066204819999999996, 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552648+0000", 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552648+0000", 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552648+0000", 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7300000000000002e-07 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552685+0000", 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6899e-05 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552699+0000", 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3518e-05 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552709+0000", 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8970000000000003e-06 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552791+0000", 2024-09-16T03:41:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1874999999999998e-05 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552797+0000", 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7610000000000002e-06 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559231+0000", 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064338989999999999 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559269+0000", 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7360000000000001e-05 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.7s2 84/55 ECSubWrite(tid=76, reqid=osd.9.0:75, at_version=0'0, trim_to=0'0, roll_forward_to=84'58))", 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:03.499100+0000", 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.7782099700000007, 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085907650000000002, 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499100+0000", 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499100+0000", 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499100+0000", 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6000000000000002e-07 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499112+0000", 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1156e-05 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499127+0000", 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5437000000000001e-05 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499136+0000", 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2159999999999995e-06 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499207+0000", 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1076999999999997e-05 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499218+0000", 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0662999999999999e-05 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.507649+0000", 2024-09-16T03:41:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084310819999999995 2024-09-16T03:41:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.507690+0000", 2024-09-16T03:41:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1273999999999998e-05 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 84/55 ECSubWrite(tid=296, reqid=osd.0.0:295, at_version=0'0, trim_to=0'0, roll_forward_to=84'77))", 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:03.499649+0000", 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.7776606099999999, 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080747600000000003, 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499649+0000", 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499649+0000", 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499650+0000", 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7999999999999993e-07 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499667+0000", 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7312e-05 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499681+0000", 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3505e-05 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499690+0000", 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2459999999999992e-06 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499762+0000", 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1555000000000002e-05 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.499777+0000", 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5704000000000001e-05 2024-09-16T03:41:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.507695+0000", 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0079173319999999991 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:03.507724+0000", 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9125999999999999e-05 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 92/31 ECSubWrite(tid=225, reqid=client.4615.0:970, at_version=92'180, trim_to=0'0, roll_forward_to=91'178))", 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:11.506965+0000", 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.77034497099999999, 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078540880000000004, 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:11.506965+0000", 2024-09-16T03:41:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:11.506965+0000", 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:11.506965+0000", 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:11.508626+0000", 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001661187 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:11.508637+0000", 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1072e-05 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:11.508646+0000", 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2e-06 2024-09-16T03:41:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:11.508723+0000", 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:11.508733+0000", 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0399e-05 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:11.514790+0000", 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060569550000000002 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:11.514819+0000", 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8532999999999999e-05 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.378 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:12.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:41:12.381 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:12.378+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:12.480 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:12.479+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:12.482 INFO:tasks.rados.rados.0.smithi121.stdout:362: done (5 left) 2024-09-16T03:41:12.482 INFO:tasks.rados.rados.0.smithi121.stdout:363: done (4 left) 2024-09-16T03:41:12.482 INFO:tasks.rados.rados.0.smithi121.stdout:364: done (3 left) 2024-09-16T03:41:12.482 INFO:tasks.rados.rados.0.smithi121.stdout:365: done (2 left) 2024-09-16T03:41:12.482 INFO:tasks.rados.rados.0.smithi121.stdout:366: done (1 left) 2024-09-16T03:41:12.482 INFO:tasks.rados.rados.0.smithi121.stdout:367: done (0 left) 2024-09-16T03:41:12.482 INFO:tasks.rados.rados.0.smithi121.stdout:368: copy_from oid 43 from oid 50 current snap is 36 2024-09-16T03:41:12.482 INFO:tasks.rados.rados.0.smithi121.stdout:369: snap_remove snap 15 2024-09-16T03:41:12.526 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:12.527 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:12.527 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:41:12.527 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:12.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:41:12.582 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:41:12.580+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:41:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:12.678 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:12.678 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:41:12.678 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:41:12.678 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:12.682 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:12.681+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:12.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:41:12.783 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:12.782+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:12.789 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4341.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.526585114, 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59293511399999999, 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:12.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4341", 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260707+0000", 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.66e-06 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260713+0000", 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6150000000000002e-06 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260728+0000", 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5333e-05 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260741+0000", 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260825+0000", 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4147999999999995e-05 2024-09-16T03:41:12.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260836+0000", 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0351e-05 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852485+0000", 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59164876099999997 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852573+0000", 2024-09-16T03:41:12.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8352999999999999e-05 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852639+0000", 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853142+0000", 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050253099999999996 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853585+0000", 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000443666 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853615+0000", 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9774e-05 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853640+0000", 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4924999999999999e-05 2024-09-16T03:41:12.796 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:483 3.8s0 3:1676b3ea:::smithi12138290-33:head [copy-from ver 28] snapc 13=[13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:42.473301+0000", 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.313989125999999, 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021928366000000001, 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:12.797 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 483 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473301+0000", 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473301+0000", 2024-09-16T03:41:12.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473303+0000", 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1009999999999999e-06 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473315+0000", 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473363+0000", 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7976000000000003e-05 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473371+0000", 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8869999999999998e-06 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473437+0000", 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5574999999999995e-05 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473444+0000", 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6089999999999997e-06 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.475559+0000", 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021149239999999998 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.475702+0000", 2024-09-16T03:41:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014371699999999999 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.492468+0000", 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016765841 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.494270+0000", 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018016639999999999 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.495153+0000", 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000883434 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.495229+0000", 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5977999999999997e-05 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:484 3.8s0 3:1676b3ea:::smithi12138290-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:42.473483+0000", 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.313806844999998, 2024-09-16T03:41:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020673568, 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 484 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473483+0000", 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473483+0000", 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473484+0000", 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.850000000000001e-07 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473489+0000", 2024-09-16T03:41:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9409999999999998e-06 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473498+0000", 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9339999999999995e-06 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473505+0000", 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9070000000000001e-06 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473736+0000", 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000231179 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473743+0000", 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.861e-06 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.476480+0000", 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027373520000000002 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.476532+0000", 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2021000000000002e-05 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.493932+0000", 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017399465999999999 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.494046+0000", 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114605 2024-09-16T03:41:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.494157+0000", 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011031699999999999 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.7s4 78/55 ECSubWrite(tid=65, reqid=osd.9.0:65, at_version=78'48, trim_to=0'0, roll_forward_to=76'46))", 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.550309+0000", 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.236980459, 2024-09-16T03:41:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015216214, 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550309+0000", 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550309+0000", 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550310+0000", 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550332+0000", 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1905000000000001e-05 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550341+0000", 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0759999999999992e-06 2024-09-16T03:41:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550350+0000", 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5010000000000004e-06 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550461+0000", 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011110199999999999 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550471+0000", 2024-09-16T03:41:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0351e-05 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565488+0000", 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015016715999999999 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565526+0000", 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8003999999999997e-05 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs2 78/31 ECSubWrite(tid=123, reqid=osd.10.0:123, at_version=78'60, trim_to=0'0, roll_forward_to=76'54))", 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.550551+0000", 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.236738517999999, 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014556872, 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550551+0000", 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550551+0000", 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550552+0000", 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.01e-07 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550576+0000", 2024-09-16T03:41:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4239e-05 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550585+0000", 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8780000000000004e-06 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550593+0000", 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8499999999999994e-06 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550809+0000", 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000216154 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550820+0000", 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1032e-05 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565062+0000", 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014242322 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565108+0000", 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5995999999999998e-05 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs3 78/55 ECSubWrite(tid=232, reqid=osd.0.0:232, at_version=78'45, trim_to=0'0, roll_forward_to=76'43))", 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.551319+0000", 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.235970654999999, 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014559028999999999, 2024-09-16T03:41:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551319+0000", 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551319+0000", 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551320+0000", 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551348+0000", 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8458000000000001e-05 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551812+0000", 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046333000000000002 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551823+0000", 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551934+0000", 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111041 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551943+0000", 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2159999999999995e-06 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565839+0000", 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013895573 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565878+0000", 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9416999999999999e-05 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs3 78/55 ECSubWrite(tid=233, reqid=osd.0.0:233, at_version=78'47, trim_to=0'0, roll_forward_to=76'43))", 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.552032+0000", 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.235258135, 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016044706999999998, 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552032+0000", 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552032+0000", 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552033+0000", 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.78e-07 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552057+0000", 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3810000000000001e-05 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552069+0000", 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2051e-05 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552078+0000", 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3559999999999999e-06 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552594+0000", 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051604000000000005 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552603+0000", 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1880000000000008e-06 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568033+0000", 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015429481 2024-09-16T03:41:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568077+0000", 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3803000000000001e-05 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s4 78/31 ECSubWrite(tid=119, reqid=osd.14.0:119, at_version=78'103, trim_to=0'0, roll_forward_to=77'101))", 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.552465+0000", 2024-09-16T03:41:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.234825214000001, 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015710608000000001, 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552465+0000", 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552465+0000", 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552465+0000", 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1899999999999998e-07 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552493+0000", 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7671e-05 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552503+0000", 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7610000000000002e-06 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552511+0000", 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8920000000000006e-06 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552597+0000", 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5118999999999996e-05 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552605+0000", 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9109999999999993e-06 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568142+0000", 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015537268 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568175+0000", 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3566999999999999e-05 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s5 78/31 ECSubWrite(tid=131, reqid=osd.1.0:131, at_version=78'54, trim_to=0'0, roll_forward_to=76'52))", 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.553878+0000", 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.233412438, 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014243021999999999, 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553878+0000", 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553878+0000", 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553878+0000", 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4699999999999998e-07 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553938+0000", 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0383000000000001e-05 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553954+0000", 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5807999999999998e-05 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553964+0000", 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6020000000000006e-06 2024-09-16T03:41:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554032+0000", 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8257999999999996e-05 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554039+0000", 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4699999999999996e-06 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568066+0000", 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014026733 2024-09-16T03:41:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568121+0000", 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4521000000000002e-05 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.9s1 78/55 ECSubWrite(tid=132, reqid=osd.1.0:132, at_version=78'64, trim_to=0'0, roll_forward_to=77'62))", 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.553990+0000", 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.233300311000001, 2024-09-16T03:41:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014899674, 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553990+0000", 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553990+0000", 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553990+0000", 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2900000000000004e-07 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554010+0000", 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9592000000000001e-05 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554018+0000", 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.636e-06 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554026+0000", 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4440000000000002e-06 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554096+0000", 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0170000000000001e-05 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554103+0000", 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9179999999999997e-06 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568859+0000", 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014756132999999999 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568889+0000", 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0252000000000001e-05 2024-09-16T03:41:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s3 78/31 ECSubWrite(tid=139, reqid=osd.6.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.554150+0000", 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.233140193000001, 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014696384, 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554150+0000", 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554150+0000", 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554151+0000", 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:41:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554162+0000", 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1537e-05 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554177+0000", 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4235e-05 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554185+0000", 2024-09-16T03:41:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1790000000000008e-06 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554433+0000", 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024847500000000001 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554444+0000", 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.039e-05 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568773+0000", 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014329328000000001 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568846+0000", 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3178999999999998e-05 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.es1 78/55 ECSubWrite(tid=37, reqid=osd.3.0:37, at_version=78'55, trim_to=0'0, roll_forward_to=76'53))", 2024-09-16T03:41:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555245+0000", 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.232044474, 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012957177, 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555245+0000", 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555245+0000", 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555246+0000", 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5600000000000001e-07 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555524+0000", 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027771099999999998 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555540+0000", 2024-09-16T03:41:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6900000000000001e-05 2024-09-16T03:41:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555550+0000", 2024-09-16T03:41:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4879999999999996e-06 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555682+0000", 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013197099999999999 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555693+0000", 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0667e-05 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568179+0000", 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012485978999999999 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568203+0000", 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4105e-05 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s5 78/31 ECSubWrite(tid=133, reqid=osd.1.0:133, at_version=78'56, trim_to=0'0, roll_forward_to=76'52))", 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555550+0000", 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.231739587, 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015172118, 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555550+0000", 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555550+0000", 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555551+0000", 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555575+0000", 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3643999999999999e-05 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555601+0000", 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6574e-05 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555613+0000", 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1225e-05 2024-09-16T03:41:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566114+0000", 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010501310999999999 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566128+0000", 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4099e-05 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570627+0000", 2024-09-16T03:41:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044985989999999998 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570722+0000", 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5829999999999996e-05 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.9s1 78/55 ECSubWrite(tid=134, reqid=osd.1.0:134, at_version=78'66, trim_to=0'0, roll_forward_to=77'62))", 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555670+0000", 2024-09-16T03:41:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.231619568999999, 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015094101, 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555670+0000", 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555670+0000", 2024-09-16T03:41:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555671+0000", 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3599999999999999e-07 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555733+0000", 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1939000000000004e-05 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555745+0000", 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2125e-05 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555761+0000", 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6169000000000001e-05 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566189+0000", 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010428181 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566198+0000", 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4849999999999992e-06 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570680+0000", 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044819810000000003 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570764+0000", 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4784999999999995e-05 2024-09-16T03:41:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.es1 78/55 ECSubWrite(tid=38, reqid=osd.3.0:38, at_version=78'57, trim_to=0'0, roll_forward_to=76'53))", 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555707+0000", 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.231582790999999, 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014962289, 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555707+0000", 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555707+0000", 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555708+0000", 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0270000000000001e-06 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555735+0000", 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.667e-05 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555747+0000", 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2546e-05 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555767+0000", 2024-09-16T03:41:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9230000000000001e-05 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566201+0000", 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010434173999999999 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566211+0000", 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0064999999999999e-05 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570621+0000", 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044097099999999998 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570669+0000", 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8866999999999998e-05 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s4 78/31 ECSubWrite(tid=120, reqid=osd.14.0:120, at_version=78'105, trim_to=0'0, roll_forward_to=77'101))", 2024-09-16T03:41:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.556218+0000", 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.231072017000001, 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014634856, 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556218+0000", 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556218+0000", 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556219+0000", 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0100000000000001e-06 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556243+0000", 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4111999999999999e-05 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556259+0000", 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5766000000000001e-05 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556268+0000", 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3100000000000006e-06 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566183+0000", 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099146909999999998 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566193+0000", 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0249e-05 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570674+0000", 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044805599999999998 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570853+0000", 2024-09-16T03:41:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017915800000000001 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s3 78/31 ECSubWrite(tid=140, reqid=osd.6.0:140, at_version=78'119, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.557327+0000", 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.229962907000001, 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013347019, 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557327+0000", 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557327+0000", 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557328+0000", 2024-09-16T03:41:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4799999999999997e-07 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557352+0000", 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4025000000000001e-05 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557368+0000", 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6402999999999999e-05 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557378+0000", 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9869999999999994e-06 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566126+0000", 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087474890000000007 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566136+0000", 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0261000000000001e-05 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570628+0000", 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044920439999999997 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570674+0000", 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5961999999999998e-05 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s3 78/31 ECSubWrite(tid=141, reqid=osd.6.0:141, at_version=78'121, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.559430+0000", 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.227859544999999, 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012847043000000001, 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559430+0000", 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559430+0000", 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559431+0000", 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559449+0000", 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8204000000000001e-05 2024-09-16T03:41:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559462+0000", 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2758000000000001e-05 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.559469+0000", 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5830000000000003e-06 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566857+0000", 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073887680000000004 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566867+0000", 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1309999999999995e-06 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.572243+0000", 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005376746 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.572277+0000", 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4097000000000002e-05 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:871 3.8s0 3:1676b3ea:::smithi12138290-33:head [rollback 1b] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.480941+0000", 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.306349287, 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012379338, 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 871 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480941+0000", 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480941+0000", 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480941+0000", 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6899999999999997e-07 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480945+0000", 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480963+0000", 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8241999999999999e-05 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480981+0000", 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8071e-05 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481104+0000", 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012331899999999999 2024-09-16T03:41:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481113+0000", 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2670000000000006e-06 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.488303+0000", 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071907660000000003 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.488412+0000", 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108207 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.489029+0000", 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061767600000000005 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491261+0000", 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022320730000000002 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.493239+0000", 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019773740000000001 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.493320+0000", 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1286999999999996e-05 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:872 3.8s0 3:1676b3ea:::smithi12138290-33:head [append 0~0] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.480997+0000", 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.306293181000001, 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012558303999999999, 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 872 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480997+0000", 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480997+0000", 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480997+0000", 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9500000000000003e-07 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481000+0000", 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7030000000000002e-06 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481005+0000", 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5260000000000001e-06 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481010+0000", 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4499999999999997e-06 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481152+0000", 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014221899999999999 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481159+0000", 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0260000000000002e-06 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.489506+0000", 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083472319999999996 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.489577+0000", 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1041e-05 2024-09-16T03:41:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490184+0000", 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060687600000000001 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491264+0000", 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00108007 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.493457+0000", 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021929649999999998 2024-09-16T03:41:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.493555+0000", 2024-09-16T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7701000000000005e-05 2024-09-16T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:12.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:41:12.893 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:12.891+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:12.993 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:12.992+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:13.037 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:13.037 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:13.037 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:41:13.037 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:13.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:41:13.094 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:13.092+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:13.148 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:13.148 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:41:13.148 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:41:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:13.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:41:13.201 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:13.199+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:13.260 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:375 3.3s0 3:c98c73f3:::smithi12138290-40:head [append 0~671744] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.720795592000002, 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086953299999999994, 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 375 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:41:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537291+0000", 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.59e-06 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537708+0000", 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041628699999999999 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537721+0000", 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3738e-05 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537728+0000", 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4409999999999998e-06 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537798+0000", 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0543000000000005e-05 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537931+0000", 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013217700000000001 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.539173+0000", 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012421660000000001 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.540584+0000", 2024-09-16T03:41:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001411673 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545889+0000", 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053041340000000003 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545985+0000", 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6581000000000002e-05 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:376 3.3s0 3:c98c73f3:::smithi12138290-40:head [append 0~770048] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.716079331000003, 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087274550000000003, 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 376 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542008+0000", 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.756e-06 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542428+0000", 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041977300000000002 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542440+0000", 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2894000000000001e-05 2024-09-16T03:41:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542447+0000", 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.584e-06 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542503+0000", 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5483999999999999e-05 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542630+0000", 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012754500000000001 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.543864+0000", 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012343090000000001 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545743+0000", 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018785799999999999 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550638+0000", 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048951790000000004 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550733+0000", 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5351000000000002e-05 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:377 3.3s0 3:c98c73f3:::smithi12138290-40:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.715623262999998, 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084428420000000007, 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 377 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542463+0000", 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2600000000000002e-07 2024-09-16T03:41:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542467+0000", 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6040000000000003e-06 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542473+0000", 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6019999999999999e-06 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542476+0000", 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.393e-06 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.544778+0000", 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023019329999999999 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.544876+0000", 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7448000000000006e-05 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545337+0000", 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046103399999999998 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.547435+0000", 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020983320000000001 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550831+0000", 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033956920000000001 2024-09-16T03:41:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550905+0000", 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4078000000000005e-05 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:378 3.3s0 3:c98c73f3:::smithi12138290-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.715586840999997, 2024-09-16T03:41:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011059941, 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 378 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1699999999999998e-07 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542502+0000", 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8040000000000002e-06 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542509+0000", 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3320000000000003e-06 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542515+0000", 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.49e-06 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.546046+0000", 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035316670000000001 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.546144+0000", 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8040000000000003e-05 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.551270+0000", 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005125597 2024-09-16T03:41:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.551384+0000", 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011363599999999999 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.553558+0000", 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021747580000000002 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.773278721000001, 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060036960000000002, 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:41:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484807+0000", 2024-09-16T03:41:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0699999999999997e-07 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484830+0000", 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3292999999999999e-05 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484841+0000", 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0891e-05 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484848+0000", 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8070000000000002e-06 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484897+0000", 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8887999999999997e-05 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484904+0000", 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8430000000000004e-06 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490773+0000", 2024-09-16T03:41:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058694940000000003 2024-09-16T03:41:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490810+0000", 2024-09-16T03:41:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6973e-05 2024-09-16T03:41:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=93, reqid=osd.10.0:93, at_version=73'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.772495385999999, 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056680849999999998, 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485590+0000", 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5599999999999999e-07 2024-09-16T03:41:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485609+0000", 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9238000000000001e-05 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485620+0000", 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0781000000000001e-05 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485630+0000", 2024-09-16T03:41:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7270000000000002e-06 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485684+0000", 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4517e-05 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.485695+0000", 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0542e-05 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491225+0000", 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055304880000000001 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491258+0000", 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2635999999999997e-05 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=94, reqid=osd.10.0:94, at_version=73'42, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.771307608000001, 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065321210000000001, 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486778+0000", 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0399999999999995e-07 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486805+0000", 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6245999999999999e-05 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486815+0000", 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0239e-05 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.486820+0000", 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3020000000000002e-06 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488642+0000", 2024-09-16T03:41:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001822001 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488651+0000", 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5639999999999997e-06 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493274+0000", 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046231290000000001 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493310+0000", 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5936000000000003e-05 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=95, reqid=client.4615.0:649, at_version=73'44, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.769592545999998, 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097259600000000005, 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488494+0000", 2024-09-16T03:41:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5099999999999998e-07 2024-09-16T03:41:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488724+0000", 2024-09-16T03:41:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000229764 2024-09-16T03:41:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488736+0000", 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.221e-05 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488744+0000", 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3110000000000006e-06 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492660+0000", 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039159759999999998 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492669+0000", 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3449999999999995e-06 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498200+0000", 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055302169999999996 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498219+0000", 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9185999999999999e-05 2024-09-16T03:41:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=97, reqid=client.4615.0:650, at_version=73'45, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.765143643000002, 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052967650000000002, 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492942+0000", 2024-09-16T03:41:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0399999999999996e-07 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493161+0000", 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021878300000000001 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493171+0000", 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5929999999999995e-06 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493178+0000", 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2389999999999996e-06 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493367+0000", 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000189581 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493389+0000", 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1389000000000001e-05 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498222+0000", 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048334980000000003 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498238+0000", 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6178e-05 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s4 73/55 ECSubWrite(tid=98, reqid=osd.10.0:98, at_version=73'24, trim_to=0'0, roll_forward_to=56'20))", 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.764420660999999, 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0063600940000000002, 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493665+0000", 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493680+0000", 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4543e-05 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493688+0000", 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.687e-06 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493694+0000", 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8209999999999998e-06 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493766+0000", 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2169999999999995e-05 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493776+0000", 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2890000000000003e-06 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499989+0000", 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062135269999999999 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500025+0000", 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5596999999999998e-05 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=99, reqid=client.4615.0:651, at_version=73'46, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.494942+0000", 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.763143263, 2024-09-16T03:41:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052685420000000002, 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494942+0000", 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494942+0000", 2024-09-16T03:41:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494943+0000", 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2200000000000003e-07 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495356+0000", 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041339099999999999 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495371+0000", 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4537e-05 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495379+0000", 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6829999999999997e-06 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495467+0000", 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7392000000000006e-05 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495476+0000", 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5240000000000006e-06 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500158+0000", 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046818190000000003 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500211+0000", 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2473999999999999e-05 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=100, reqid=client.4615.0:652, at_version=73'47, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.761013009999999, 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083283569999999998, 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497073+0000", 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.61e-07 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497223+0000", 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014987799999999999 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497236+0000", 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3278999999999999e-05 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497245+0000", 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8130000000000002e-06 2024-09-16T03:41:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497329+0000", 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3671999999999994e-05 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497338+0000", 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7000000000000003e-06 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.505360+0000", 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080216100000000002 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.505401+0000", 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0744000000000002e-05 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=93, reqid=osd.14.0:92, at_version=0'0, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.757947868999999, 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0079817680000000002, 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500138+0000", 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7400000000000002e-07 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500148+0000", 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037999999999999e-05 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500163+0000", 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5109999999999999e-05 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500172+0000", 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8419999999999994e-06 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500224+0000", 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1759999999999999e-05 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500233+0000", 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8359999999999998e-06 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508092+0000", 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078594029999999992 2024-09-16T03:41:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508119+0000", 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7005e-05 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=101, reqid=client.4615.0:653, at_version=73'48, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.500387+0000", 2024-09-16T03:41:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.75769815, 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077870200000000004, 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500387+0000", 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500387+0000", 2024-09-16T03:41:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500388+0000", 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500412+0000", 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4386e-05 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500421+0000", 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1020000000000003e-06 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500433+0000", 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2011000000000001e-05 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500496+0000", 2024-09-16T03:41:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500507+0000", 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0499000000000001e-05 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508155+0000", 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076484200000000004 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508174+0000", 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9011000000000001e-05 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=94, reqid=osd.14.0:94, at_version=73'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.756963550999998, 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070195409999999998, 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501122+0000", 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6499999999999999e-07 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501143+0000", 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0565e-05 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501156+0000", 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3278999999999999e-05 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501164+0000", 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0900000000000005e-06 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501227+0000", 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2682999999999996e-05 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501245+0000", 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8108e-05 2024-09-16T03:41:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508122+0000", 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068766180000000001 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508141+0000", 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9533000000000001e-05 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=102, reqid=osd.10.0:102, at_version=73'50, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.75615436, 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087707529999999992, 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501932+0000", 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3e-07 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501949+0000", 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7711e-05 2024-09-16T03:41:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501962+0000", 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2549e-05 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501969+0000", 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1720000000000001e-06 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502019+0000", 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0340000000000003e-05 2024-09-16T03:41:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502028+0000", 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8570000000000001e-06 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510672+0000", 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086435399999999999 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510702+0000", 2024-09-16T03:41:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9954000000000002e-05 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s4 73/55 ECSubWrite(tid=103, reqid=osd.10.0:98, at_version=0'0, trim_to=0'0, roll_forward_to=73'24))", 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502008+0000", 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.756077089000001, 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006105933, 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502008+0000", 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502008+0000", 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502009+0000", 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502016+0000", 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3660000000000003e-06 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502024+0000", 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.542e-06 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502030+0000", 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9379999999999997e-06 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502199+0000", 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016938200000000001 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502209+0000", 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508072+0000", 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058631170000000002 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.508114+0000", 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2546999999999998e-05 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=95, reqid=osd.14.0:95, at_version=73'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.755999590000002, 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085979139999999999, 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3099999999999998e-07 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502103+0000", 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6769000000000002e-05 2024-09-16T03:41:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502114+0000", 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1258e-05 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502121+0000", 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6429999999999998e-06 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502175+0000", 2024-09-16T03:41:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3698e-05 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502183+0000", 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6589999999999996e-06 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510650+0000", 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084665680000000007 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510684+0000", 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3887999999999998e-05 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.es6 78/55 ECSubWrite(tid=38, reqid=osd.3.0:38, at_version=78'57, trim_to=0'0, roll_forward_to=76'53))", 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555743+0000", 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.702341926999999, 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058697580000000001, 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555743+0000", 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555743+0000", 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555744+0000", 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003e-06 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555812+0000", 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7830000000000006e-05 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555841+0000", 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8619000000000002e-05 2024-09-16T03:41:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555854+0000", 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3424000000000001e-05 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556254+0000", 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039962800000000002 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556264+0000", 2024-09-16T03:41:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6020000000000006e-06 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561560+0000", 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052968650000000004 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561613+0000", 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2787000000000001e-05 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 78/31 ECSubWrite(tid=120, reqid=osd.14.0:120, at_version=78'105, trim_to=0'0, roll_forward_to=77'101))", 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.556185+0000", 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.701900041, 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054648550000000002, 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556185+0000", 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556185+0000", 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556186+0000", 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:41:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556210+0000", 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3618999999999999e-05 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556220+0000", 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0699e-05 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556229+0000", 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0739999999999993e-06 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556310+0000", 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1716000000000002e-05 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556316+0000", 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-09-16T03:41:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561619+0000", 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053029849999999996 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561650+0000", 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1433999999999998e-05 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:13.360 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:41:13.361 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:41:13.358+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:41:13.461 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:13.459+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:13.476 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:41:13.476 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:13.484 INFO:tasks.rados.rados.0.smithi121.stdout:369: done (1 left) 2024-09-16T03:41:13.484 INFO:tasks.rados.rados.0.smithi121.stdout:370: copy_from oid 15 from oid 34 current snap is 36 2024-09-16T03:41:13.484 INFO:tasks.rados.rados.0.smithi121.stdout:368: finishing copy_from racing read to smithi12138290-43 2024-09-16T03:41:13.484 INFO:tasks.rados.rados.0.smithi121.stdout:368: finishing copy_from to smithi12138290-43 2024-09-16T03:41:13.484 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 43 v 69 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:41:13.484 INFO:tasks.rados.rados.0.smithi121.stdout:368: done (1 left) 2024-09-16T03:41:13.484 INFO:tasks.rados.rados.0.smithi121.stdout:371: rollback oid 44 current snap is 36 2024-09-16T03:41:13.484 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 44 to 32 2024-09-16T03:41:13.484 INFO:tasks.rados.rados.0.smithi121.stdout:372: delete oid 46 current snap is 36 2024-09-16T03:41:13.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:41:13.490 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:41:13.494 INFO:tasks.rados.rados.0.smithi121.stdout:372: done (2 left) 2024-09-16T03:41:13.494 INFO:tasks.rados.rados.0.smithi121.stdout:373: snap_remove snap 30 2024-09-16T03:41:13.561 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:13.560+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:13.601 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:13.601 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:13.601 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:41:13.601 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:41:13.601 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:13.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:41:13.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:41:13.662 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:13.661+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:13.727 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:41:13.727 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:41:13.744 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:41:13.744 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:41:13.754 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:13.755 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:41:13.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:41:13.755 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-09-16T03:41:13.755 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:13.761 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:41:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:41:13.762 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:13.761+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:13.763 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:41:13.778 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:41:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:41:13.794 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:41:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:41:13.811 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:41:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:41:13.828 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:41:13.828 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:41:13.845 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:41:13.845 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:41:13.861 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:41:13.862 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:41:13.863 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:13.862+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:13.869 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:13.869 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:13.878 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:41:13.879 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:41:13.880 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:41:13.895 INFO:teuthology.orchestra.run.smithi076.stdout:get_pg_num_history pg_num_history pg_num_history(e83 pg_nums {1={24=1},2={24=8},3={31=16}} deleted_pools ) 2024-09-16T03:41:13.895 INFO:teuthology.orchestra.run.smithi076.stdout:Importing pgid 3.cs2 2024-09-16T03:41:13.895 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:32badfe7:::smithi12138290-17:15# 2024-09-16T03:41:13.895 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:32badfe7:::smithi12138290-17:19# 2024-09-16T03:41:13.896 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:32badfe7:::smithi12138290-17:head# 2024-09-16T03:41:13.897 INFO:teuthology.orchestra.run.smithi076.stdout:snapset 19=[]:{15=[11,b,8],19=[19]} 2024-09-16T03:41:13.898 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:335b675a:::smithi12138290-38:e# 2024-09-16T03:41:13.898 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:41:13.898 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:41:13.899 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:335b675a:::smithi12138290-38:head# 2024-09-16T03:41:13.899 INFO:teuthology.orchestra.run.smithi076.stdout:snapset e=[]:{e=[b,8]} 2024-09-16T03:41:13.899 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:3432dbf3:::smithi12138290-44:11# 2024-09-16T03:41:13.899 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:3432dbf3:::smithi12138290-44:1b# 2024-09-16T03:41:13.900 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:3432dbf3:::smithi12138290-44:head# 2024-09-16T03:41:13.900 INFO:teuthology.orchestra.run.smithi076.stdout:snapset 1b=[]:{11=[11,b,8],1b=[1b,1a,19]} 2024-09-16T03:41:13.900 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:38c596e1:::smithi12138290-26:11# 2024-09-16T03:41:13.901 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:38c596e1:::smithi12138290-26:19# 2024-09-16T03:41:13.902 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:38c596e1:::smithi12138290-26:1b# 2024-09-16T03:41:13.902 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:38c596e1:::smithi12138290-26:head# 2024-09-16T03:41:13.902 INFO:teuthology.orchestra.run.smithi076.stdout:snapset 1c=[]:{11=[11],19=[19],1b=[1b,1a]} 2024-09-16T03:41:13.904 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:39b8e90f:::smithi12138290-2:17# 2024-09-16T03:41:13.904 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:39b8e90f:::smithi12138290-2:1a# 2024-09-16T03:41:13.906 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:39b8e90f:::smithi12138290-2:head# 2024-09-16T03:41:13.906 INFO:teuthology.orchestra.run.smithi076.stdout:snapset 1a=[]:{17=[11,b,8],1a=[1a,19]} 2024-09-16T03:41:13.907 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:3aac74e6:::smithi12138290-45:d# 2024-09-16T03:41:13.908 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:3aac74e6:::smithi12138290-45:head# 2024-09-16T03:41:13.909 INFO:teuthology.orchestra.run.smithi076.stdout:snapset d=[]:{d=[b,8]} 2024-09-16T03:41:13.909 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:3e925a6e:::smithi12138290-43:1b# 2024-09-16T03:41:13.910 INFO:teuthology.orchestra.run.smithi076.stdout:Write 2#3:3e925a6e:::smithi12138290-43:head# 2024-09-16T03:41:13.910 INFO:teuthology.orchestra.run.smithi076.stdout:snapset 1b=[]:{1b=[1b,1a,19,11,b,8]} 2024-09-16T03:41:13.911 INFO:teuthology.orchestra.run.smithi076.stdout:write_pg epoch 82 info 3.cs2( v 82'155 (0'0,82'155] local-lis/les=31/32 n=19 ec=31/31 lis/c=31/31 les/c/f=32/32/0 sis=31) 2024-09-16T03:41:13.912 INFO:teuthology.orchestra.run.smithi076.stdout:Import successful 2024-09-16T03:41:13.918 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:41:13.918 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:41:13.918 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-16T03:41:13.935 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:41:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:41:13.953 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:41:13.953 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:41:13.964 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:41:13.963+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:41:13.970 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:41:13.970 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:41:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:41:13.973 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:41:13.989 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:13.989 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:13.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:41:14.065 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:14.064+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:14.141 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:14.141 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:14.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:41:14.166 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:14.164+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:14.252 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:14.253 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:14.261 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:41:14.267 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:14.265+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:14.365 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:14.365 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:14.367 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:14.366+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:14.374 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:41:14.429 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /var/log/ceph/exp.3.cs2.12 2024-09-16T03:41:14.455 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.12 2024-09-16T03:41:14.455 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-12 show-label 2024-09-16T03:41:14.479 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:14.479 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:14.480 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:14.478+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:14.481 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T03:41:14.484 INFO:tasks.rados.rados.0.smithi121.stdout:373: done (2 left) 2024-09-16T03:41:14.484 INFO:tasks.rados.rados.0.smithi121.stdout:374: snap_create 2024-09-16T03:41:14.484 INFO:tasks.rados.rados.0.smithi121.stdout:371: finishing rollback tid 1 to smithi12138290-44 2024-09-16T03:41:14.484 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 44 v 116 (ObjNum 161 snap 0 seq_num 1) dirty dne 2024-09-16T03:41:14.484 INFO:tasks.rados.rados.0.smithi121.stdout:370: finishing copy_from racing read to smithi12138290-15 2024-09-16T03:41:14.484 INFO:tasks.rados.rados.0.smithi121.stdout:370: finishing copy_from to smithi12138290-15 2024-09-16T03:41:14.485 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 15 v 65 (ObjNum 153 snap 31 seq_num 153) dirty exists 2024-09-16T03:41:14.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:41:14.528 INFO:teuthology.orchestra.run.smithi076.stdout:inferring bluefs devices from bluestore path 2024-09-16T03:41:14.568 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:14.567+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:14.633 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:14.633 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:14.642 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:41:14.669 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:14.668+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:14.744 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:14.745 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:14.753 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:41:14.770 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:41:14.769+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:41:14.810 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "/var/lib/ceph/osd/ceph-12/block": { 2024-09-16T03:41:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_uuid": "a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1", 2024-09-16T03:41:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 96636764160, 2024-09-16T03:41:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "btime": "2024-09-16T03:38:35.182493+0000", 2024-09-16T03:41:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "main", 2024-09-16T03:41:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "bfm_blocks": "23592960", 2024-09-16T03:41:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "bfm_blocks_per_key": "128", 2024-09-16T03:41:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "bfm_bytes_per_block": "4096", 2024-09-16T03:41:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "bfm_size": "96636764160", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs": "1", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_fsid": "ecf7a42a-e941-4b24-8693-0ff8b6eb41b1", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_version_when_created": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "created_at": "2024-09-16T03:38:38.065370Z", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "elastic_shared_blobs": "1", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": "16", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "kv_backend": "rocksdb", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "magic": "ceph osd volume v026", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "multi": "yes", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "ready": "ready", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "19", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bluestore", 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": "12" 2024-09-16T03:41:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T03:41:14.812 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:14.813 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.12 2024-09-16T03:41:14.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-12 show-sharding 2024-09-16T03:41:14.859 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:14.859 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:14.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:41:14.872 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:14.870+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:14.973 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:41:14.971+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:41:14.983 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:14.984 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:14.992 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:41:15.073 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:15.071+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:15.097 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:15.097 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:15.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:41:15.174 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:15.172+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:15.210 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:15.210 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:15.219 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:41:15.275 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:15.273+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:15.323 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:15.324 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:15.332 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:41:15.375 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:15.374+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:15.476 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:15.476 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:15.476 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:15.474+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:15.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:41:15.501 INFO:tasks.rados.rados.0.smithi121.stdout:370: done (2 left) 2024-09-16T03:41:15.501 INFO:tasks.rados.rados.0.smithi121.stdout:371: done (1 left) 2024-09-16T03:41:15.501 INFO:tasks.rados.rados.0.smithi121.stdout:374: done (0 left) 2024-09-16T03:41:15.502 INFO:tasks.rados.rados.0.smithi121.stdout:375: rollback oid 4 current snap is 37 2024-09-16T03:41:15.502 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 4 to 27 2024-09-16T03:41:15.502 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 29 current snap is 37 2024-09-16T03:41:15.502 INFO:tasks.rados.rados.0.smithi121.stdout:376: seq_num 160 ranges {2293760=294912} 2024-09-16T03:41:15.516 INFO:tasks.rados.rados.0.smithi121.stdout:376: writing smithi12138290-29 from 2293760 to 2588672 tid 1 2024-09-16T03:41:15.516 INFO:tasks.rados.rados.0.smithi121.stdout:377: snap_remove snap 9 2024-09-16T03:41:15.577 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:41:15.576+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:41:15.590 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:15.590 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:15.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:41:15.678 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:15.677+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:15.702 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:15.702 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:15.711 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:41:15.779 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:15.777+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:15.855 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:15.856 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:15.864 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:41:15.878 INFO:teuthology.orchestra.run.smithi076.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-09-16T03:41:15.882 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.12 2024-09-16T03:41:15.882 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-09-16T03:41:15.882 INFO:tasks.thrashosds.thrasher:applying new sharding: m(2) L(2,8-) 2024-09-16T03:41:15.882 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-12 --sharding 'm(2) L(2,8-)' reshard 2024-09-16T03:41:15.908 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:15.907+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:15.968 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:15.968 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:15.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:41:15.976 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:41:15.981 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:15.979+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:16.081 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:16.080+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:16.093 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:16.093 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:16.095 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:41:16.095 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:41:16.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:41:16.113 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:41:16.113 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:41:16.130 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:41:16.130 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:41:16.147 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:41:16.148 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:41:16.163 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:41:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:41:16.180 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:41:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:41:16.183 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:16.181+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:16.196 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:41:16.197 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:41:16.207 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:16.207 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:16.207 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:41:16.207 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:41:16.207 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:16.214 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:41:16.214 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:41:16.216 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:41:16.230 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:41:16.230 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:41:16.248 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:41:16.248 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:41:16.265 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:41:16.265 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:41:16.282 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:41:16.282 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:41:16.282 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-16T03:41:16.283 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:16.282+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:16.299 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:41:16.299 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:41:16.315 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:41:16.315 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:41:16.321 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=31, reqid=osd.8.0:31, at_version=49'22, trim_to=0'0, roll_forward_to=46'18))", 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 47.823734832, 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059536579999999997, 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495458+0000", 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495459+0000", 2024-09-16T03:41:16.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3799999999999996e-07 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495484+0000", 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5601000000000001e-05 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495505+0000", 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0460999999999999e-05 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495514+0000", 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1390000000000001e-06 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495592+0000", 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7877e-05 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495607+0000", 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5576000000000001e-05 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501362+0000", 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057546280000000003 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501411+0000", 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9438000000000003e-05 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.326 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 43.814445395, 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068498659999999996, 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504748+0000", 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4499999999999997e-07 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504768+0000", 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0582999999999999e-05 2024-09-16T03:41:16.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504782+0000", 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3883000000000001e-05 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504790+0000", 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3389999999999993e-06 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504921+0000", 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013093800000000001 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504930+0000", 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7499999999999992e-06 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511566+0000", 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066358449999999996 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511597+0000", 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0883000000000002e-05 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 53/31 ECSubWrite(tid=20, reqid=client.4615.0:376, at_version=53'13, trim_to=0'0, roll_forward_to=37'10))", 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.544496+0000", 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 43.774695952000002, 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059495989999999999, 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.331 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.332 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544496+0000", 2024-09-16T03:41:16.332 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544496+0000", 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.544498+0000", 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3519999999999999e-06 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545106+0000", 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00060862000000000004 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.333 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545125+0000", 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9052999999999999e-05 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545136+0000", 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0125e-05 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545201+0000", 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5769e-05 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.545212+0000", 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0611000000000001e-05 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550406+0000", 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051939519999999999 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.550446+0000", 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0117000000000002e-05 2024-09-16T03:41:16.335 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 63/31 ECSubWrite(tid=140, reqid=osd.0.0:140, at_version=63'56, trim_to=0'0, roll_forward_to=60'52))", 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.476911+0000", 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 33.842281292000003, 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0063366669999999998, 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.336 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476911+0000", 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476911+0000", 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476912+0000", 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4099999999999998e-07 2024-09-16T03:41:16.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476932+0000", 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0027e-05 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476942+0000", 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7429999999999997e-06 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476950+0000", 2024-09-16T03:41:16.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3510000000000002e-06 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.478026+0000", 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001076411 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.478036+0000", 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5750000000000007e-06 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483217+0000", 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051814000000000001 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483248+0000", 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0419000000000001e-05 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 63/31 ECSubWrite(tid=31, reqid=osd.15.0:31, at_version=63'24, trim_to=0'0, roll_forward_to=60'22))", 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:41:16.340 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 33.842257619999998, 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0062644750000000003, 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6899999999999999e-07 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476965+0000", 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9714999999999999e-05 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.342 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476988+0000", 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3297000000000001e-05 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476998+0000", 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0254999999999999e-05 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.477097+0000", 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8939999999999998e-05 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.343 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.477108+0000", 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.039e-05 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483157+0000", 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060496719999999999 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483199+0000", 2024-09-16T03:41:16.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1937000000000002e-05 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 73/31 ECSubWrite(tid=41, reqid=osd.15.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=73'31))", 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.497839+0000", 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 23.821352979, 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060867669999999999, 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497839+0000", 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497839+0000", 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497840+0000", 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0100000000000001e-06 2024-09-16T03:41:16.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497854+0000", 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3358e-05 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497871+0000", 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6994999999999999e-05 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497880+0000", 2024-09-16T03:41:16.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3589999999999997e-06 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497967+0000", 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7106000000000003e-05 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497978+0000", 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0370999999999999e-05 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.503877+0000", 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058990930000000002 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.503926+0000", 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9475000000000003e-05 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 74/55 ECSubWrite(tid=30, reqid=osd.3.0:30, at_version=74'47, trim_to=0'0, roll_forward_to=73'39))", 2024-09-16T03:41:16.349 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.535733+0000", 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 22.783459699000002, 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066050730000000004, 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535733+0000", 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535733+0000", 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535734+0000", 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.246e-06 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535760+0000", 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5706000000000001e-05 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535773+0000", 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3682e-05 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535782+0000", 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9090000000000006e-06 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.536271+0000", 2024-09-16T03:41:16.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00048888700000000002 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.536282+0000", 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.078e-05 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542291+0000", 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060094379999999998 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.353 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542338+0000", 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6424999999999997e-05 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 74/31 ECSubWrite(tid=131, reqid=osd.12.0:131, at_version=74'68, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.539984+0000", 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 22.779208415999999, 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078357930000000006, 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.354 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539984+0000", 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539984+0000", 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539985+0000", 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0299999999999997e-07 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540008+0000", 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2765999999999999e-05 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540022+0000", 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4455e-05 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540031+0000", 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2e-06 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540256+0000", 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000224391 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540280+0000", 2024-09-16T03:41:16.357 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3952e-05 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547770+0000", 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007489941 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547820+0000", 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0185000000000001e-05 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.358 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 84/55 ECSubWrite(tid=293, reqid=osd.0.0:293, at_version=84'75, trim_to=0'0, roll_forward_to=83'71))", 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.481368+0000", 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 12.837824875000001, 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0076366239999999998, 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481368+0000", 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481368+0000", 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481368+0000", 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0299999999999999e-07 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481397+0000", 2024-09-16T03:41:16.360 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8648000000000001e-05 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481406+0000", 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5810000000000003e-06 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481414+0000", 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8499999999999994e-06 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.361 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.482165+0000", 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00075051999999999996 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.482177+0000", 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2602999999999999e-05 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.488963+0000", 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0067854270000000001 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.362 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489004+0000", 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1492000000000002e-05 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 84/55 ECSubWrite(tid=50, reqid=osd.3.0:50, at_version=84'71, trim_to=0'0, roll_forward_to=83'69))", 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.483239+0000", 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 12.835952927999999, 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0064962910000000004, 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.363 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483239+0000", 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483239+0000", 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.364 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483240+0000", 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000001e-07 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483267+0000", 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6974e-05 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483284+0000", 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6869999999999999e-05 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.365 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483293+0000", 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3400000000000004e-06 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483382+0000", 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8183000000000006e-05 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483392+0000", 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0691e-05 2024-09-16T03:41:16.366 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489700+0000", 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0063073249999999999 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489736+0000", 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6229999999999997e-05 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.367 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 84/31 ECSubWrite(tid=88, reqid=osd.8.0:88, at_version=84'69, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.484725+0000", 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 12.834467674000001, 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060921380000000004, 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484725+0000", 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.368 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484725+0000", 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484725+0000", 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5100000000000001e-07 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484749+0000", 2024-09-16T03:41:16.369 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3714e-05 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484760+0000", 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0857e-05 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484768+0000", 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5359999999999992e-06 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.370 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484937+0000", 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000169347 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484946+0000", 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7749999999999998e-06 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.490785+0000", 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058390350000000002 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.371 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.490817+0000", 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1622999999999999e-05 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 84/31 ECSubWrite(tid=89, reqid=osd.8.0:89, at_version=84'71, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.485262+0000", 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 12.833930416999999, 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010975762, 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.372 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485262+0000", 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485262+0000", 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.373 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485263+0000", 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485286+0000", 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3339000000000001e-05 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485300+0000", 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3658999999999999e-05 2024-09-16T03:41:16.374 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485319+0000", 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8681e-05 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489431+0000", 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041121150000000004 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.375 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489441+0000", 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8400000000000007e-06 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496195+0000", 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006754566 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496238+0000", 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2620000000000002e-05 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.376 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 84/55 ECSubWrite(tid=294, reqid=osd.0.0:293, at_version=0'0, trim_to=0'0, roll_forward_to=84'75))", 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.489541+0000", 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 12.829651815, 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0067107429999999999, 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.377 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489541+0000", 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489541+0000", 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489542+0000", 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8100000000000001e-07 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.378 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489554+0000", 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2002e-05 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489569+0000", 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5159e-05 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489578+0000", 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3770000000000002e-06 2024-09-16T03:41:16.379 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489633+0000", 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5251999999999999e-05 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489643+0000", 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6190000000000006e-06 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.380 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496201+0000", 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0065581270000000004 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496251+0000", 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0226000000000003e-05 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 84/55 ECSubWrite(tid=295, reqid=osd.0.0:295, at_version=84'77, trim_to=0'0, roll_forward_to=84'75))", 2024-09-16T03:41:16.381 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.491137+0000", 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 12.828055562999999, 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079076210000000001, 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491137+0000", 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.382 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491137+0000", 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491138+0000", 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0100000000000003e-07 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491156+0000", 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.838e-05 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.383 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491168+0000", 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2320000000000001e-05 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491174+0000", 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8189999999999997e-06 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491231+0000", 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6564000000000003e-05 2024-09-16T03:41:16.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491243+0000", 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.208e-05 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.499004+0000", 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0077610650000000002 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.499044+0000", 2024-09-16T03:41:16.385 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0491999999999998e-05 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 94/31 ECSubWrite(tid=101, reqid=osd.8.0:101, at_version=94'79, trim_to=0'0, roll_forward_to=91'75))", 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.482418+0000", 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8367740509999999, 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061555799999999999, 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.386 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482418+0000", 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482418+0000", 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482419+0000", 2024-09-16T03:41:16.387 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7000000000000003e-07 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482466+0000", 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6539999999999998e-05 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482479+0000", 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3619e-05 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.388 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482492+0000", 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2337999999999999e-05 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482556+0000", 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4647000000000006e-05 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482576+0000", 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9412e-05 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.389 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488519+0000", 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059435110000000003 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488574+0000", 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4543000000000003e-05 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.390 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 94/55 ECSubWrite(tid=61, reqid=client.4615.0:985, at_version=94'79, trim_to=0'0, roll_forward_to=91'77))", 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.485387+0000", 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8338058569999998, 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0072071790000000002, 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485387+0000", 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.391 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485387+0000", 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485387+0000", 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0999999999999999e-07 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485417+0000", 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0151999999999999e-05 2024-09-16T03:41:16.392 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485427+0000", 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9660000000000008e-06 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485435+0000", 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2879999999999993e-06 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485600+0000", 2024-09-16T03:41:16.393 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000164959 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485611+0000", 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0937999999999999e-05 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.492561+0000", 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069494639999999998 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.394 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.492594+0000", 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2901999999999998e-05 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 94/85 ECSubWrite(tid=212, reqid=osd.14.0:212, at_version=94'108, trim_to=0'0, roll_forward_to=91'104))", 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.485765+0000", 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.833427661, 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060379489999999999, 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.395 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485765+0000", 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485765+0000", 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.396 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485766+0000", 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0130000000000001e-06 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485791+0000", 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5695999999999999e-05 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485811+0000", 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9462000000000001e-05 2024-09-16T03:41:16.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485821+0000", 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0241000000000001e-05 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485880+0000", 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8477999999999998e-05 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485897+0000", 2024-09-16T03:41:16.398 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7427000000000001e-05 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.491766+0000", 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058688880000000001 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.491803+0000", 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6743999999999999e-05 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.399 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.401 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.401 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 94/55 ECSubWrite(tid=63, reqid=osd.3.0:63, at_version=94'83, trim_to=0'0, roll_forward_to=91'77))", 2024-09-16T03:41:16.401 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.488436+0000", 2024-09-16T03:41:16.401 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8307566980000001, 2024-09-16T03:41:16.401 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061141540000000001, 2024-09-16T03:41:16.401 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488436+0000", 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488436+0000", 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.402 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488437+0000", 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5799999999999998e-07 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488461+0000", 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4178000000000001e-05 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488473+0000", 2024-09-16T03:41:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1951e-05 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488482+0000", 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9160000000000007e-06 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.490000+0000", 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001518499 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.404 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.490009+0000", 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8210000000000008e-06 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.494509+0000", 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044997570000000001 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.494550+0000", 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1174000000000003e-05 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.405 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 94/55 ECSubWrite(tid=64, reqid=osd.3.0:63, at_version=0'0, trim_to=0'0, roll_forward_to=94'83))", 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.497633+0000", 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8215595360000001, 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059829410000000003, 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497633+0000", 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497633+0000", 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497634+0000", 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4e-07 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.407 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497649+0000", 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5420000000000001e-05 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497665+0000", 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5764999999999999e-05 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497675+0000", 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6099999999999995e-06 2024-09-16T03:41:16.408 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497740+0000", 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5225e-05 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497750+0000", 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0360000000000001e-05 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.409 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.503574+0000", 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058242220000000004 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.503616+0000", 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1399000000000001e-05 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 94/31 ECSubWrite(tid=103, reqid=client.4615.0:982, at_version=94'81, trim_to=0'0, roll_forward_to=94'79))", 2024-09-16T03:41:16.410 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.519271+0000", 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.7999211920000002, 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090919550000000005, 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.519271+0000", 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.411 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.519271+0000", 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.519272+0000", 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7700000000000004e-07 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.524471+0000", 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051987270000000002 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.412 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.524487+0000", 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5940999999999999e-05 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.524495+0000", 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.49e-06 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.524550+0000", 2024-09-16T03:41:16.413 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5198000000000002e-05 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.524559+0000", 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.528322+0000", 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037636060000000001 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.414 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.528363+0000", 2024-09-16T03:41:16.415 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0836e-05 2024-09-16T03:41:16.415 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.415 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.415 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.415 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.415 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.415 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:16.416 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:41:16.417 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:41:16.417 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:41:16.418 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:41:16.418 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:41:16.418 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:16.414+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:16.483 INFO:tasks.rados.rados.0.smithi121.stdout:377: done (2 left) 2024-09-16T03:41:16.483 INFO:tasks.rados.rados.0.smithi121.stdout:378: rollback oid 22 current snap is 37 2024-09-16T03:41:16.483 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 22 to 32 2024-09-16T03:41:16.483 INFO:tasks.rados.rados.0.smithi121.stdout:375: finishing rollback tid 0 to smithi12138290-4 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:375: finishing rollback tid 1 to smithi12138290-4 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:375: finishing rollback tid 2 to smithi12138290-4 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 4 v 87 (ObjNum 79 snap 11 seq_num 79) dirty exists 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:376: finishing write tid 1 to smithi12138290-29 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:376: finishing write tid 2 to smithi12138290-29 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:376: finishing write tid 3 to smithi12138290-29 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:376: oid 29 updating version 0 to 114 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:376: oid 29 updating version 114 to 115 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 29 v 115 (ObjNum 160 snap 37 seq_num 160) dirty exists 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:376: left oid 29 (ObjNum 160 snap 37 seq_num 160) 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:375: done (2 left) 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:376: done (1 left) 2024-09-16T03:41:16.484 INFO:tasks.rados.rados.0.smithi121.stdout:379: rollback oid 18 current snap is 37 2024-09-16T03:41:16.485 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 18 to 27 2024-09-16T03:41:16.485 INFO:tasks.rados.rados.0.smithi121.stdout:380: snap_remove snap 31 2024-09-16T03:41:16.505 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:41:16.517 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:16.516+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:16.540 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:16.540 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:16.540 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:41:16.540 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:16.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:41:16.567 INFO:teuthology.orchestra.run.smithi076.stdout:reshard success 2024-09-16T03:41:16.618 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:16.617+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:16.653 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:16.654 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:16.654 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:41:16.654 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:41:16.654 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:16.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:41:16.719 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:16.717+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:301 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~491520] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 48.320304518, 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013883662999999999, 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.806 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 301 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482860+0000", 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.911e-06 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483803+0000", 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00094328700000000001 2024-09-16T03:41:16.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483820+0000", 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6249e-05 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483828+0000", 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.704e-06 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483895+0000", 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6839000000000002e-05 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483902+0000", 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.485797+0000", 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018946049999999999 2024-09-16T03:41:16.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.485916+0000", 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011867 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487208+0000", 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012921110000000001 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493186+0000", 2024-09-16T03:41:16.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059786010000000001 2024-09-16T03:41:16.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:16.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496639+0000", 2024-09-16T03:41:16.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034530749999999999 2024-09-16T03:41:16.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496742+0000", 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102394 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:302 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~770048] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 48.316025244999999, 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014729638999999999, 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.813 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 302 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487139+0000", 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.378e-06 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488116+0000", 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00097674700000000003 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488127+0000", 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1817999999999999e-05 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488135+0000", 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6620000000000005e-06 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489443+0000", 2024-09-16T03:41:16.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001308303 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489564+0000", 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000120664 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490525+0000", 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00096115100000000004 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493256+0000", 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027313429999999998 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501784+0000", 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0085277840000000001 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501867+0000", 2024-09-16T03:41:16.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2788999999999996e-05 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:303 3.1s0 3:8a735ecd:::smithi12138290-30:head [setxattr _header (57) in=7b] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 48.315013708999999, 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013931318, 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.819 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 303 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:41:16.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488150+0000", 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000001e-07 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488152+0000", 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.509e-06 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488157+0000", 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5739999999999997e-06 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488160+0000", 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.349e-06 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491536+0000", 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033756799999999998 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491638+0000", 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010186799999999999 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492139+0000", 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050129999999999999 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493257+0000", 2024-09-16T03:41:16.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001118551 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501989+0000", 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087319819999999992 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502080+0000", 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0827000000000003e-05 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.824 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:304 3.1s0 3:8a735ecd:::smithi12138290-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 48.314996731000001, 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.024338029000000001, 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.825 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 304 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.53e-07 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488169+0000", 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3309999999999998e-06 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488173+0000", 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:41:16.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488176+0000", 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0539999999999999e-06 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492435+0000", 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042594399999999998 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:16.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492534+0000", 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8558000000000001e-05 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505269+0000", 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012735076 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505321+0000", 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1977000000000001e-05 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.512504+0000", 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0071833060000000004 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:309 3.1s0 3:82a78c8d:::smithi12138290-1:head [copy-from ver 4] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 48.312337278000001, 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020962743999999998, 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.830 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 309 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490827+0000", 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490832+0000", 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4550000000000003e-06 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490845+0000", 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2951e-05 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490873+0000", 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7566999999999999e-05 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492571+0000", 2024-09-16T03:41:16.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001698359 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492650+0000", 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8858999999999998e-05 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503971+0000", 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011320805 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511333+0000", 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073624160000000001 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511735+0000", 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00040226999999999998 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511788+0000", 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2757999999999999e-05 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.835 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:310 3.1s0 3:82a78c8d:::smithi12138290-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 48.312271299000003, 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014337858, 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.836 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 310 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490892+0000", 2024-09-16T03:41:16.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.37e-07 2024-09-16T03:41:16.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490896+0000", 2024-09-16T03:41:16.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3710000000000002e-06 2024-09-16T03:41:16.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490932+0000", 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5796000000000001e-05 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490941+0000", 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8589999999999994e-06 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493109+0000", 2024-09-16T03:41:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021679310000000001 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493167+0000", 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7572999999999999e-05 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505077+0000", 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011909968999999999 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505164+0000", 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7409000000000003e-05 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505229+0000", 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5213000000000001e-05 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:593 3.1s0 3:82a78c8d:::smithi12138290-1:head [stat,delete] snapc 18=[18,17,16,15,13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T03:41:16.843 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:48.498370+0000", 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.304792781, 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012012600999999999, 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 593 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498370+0000", 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498370+0000", 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498372+0000", 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9589999999999998e-06 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498379+0000", 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9619999999999996e-06 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498396+0000", 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6801000000000001e-05 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498405+0000", 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5380000000000008e-06 2024-09-16T03:41:16.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498490+0000", 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4481999999999995e-05 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.498497+0000", 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2989999999999999e-06 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.506967+0000", 2024-09-16T03:41:16.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084698340000000007 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.507051+0000", 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3943999999999995e-05 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.507813+0000", 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076275500000000005 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.508730+0000", 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00091662999999999998 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.510309+0000", 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001578772 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:48.510382+0000", 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3625000000000001e-05 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.849 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:893 3.1s0 3:82a78c8d:::smithi12138290-1:head [append 0~16384] snapc 1d=[1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.506642+0000", 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.296520442, 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.026920999000000001, 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.850 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 893 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506642+0000", 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506642+0000", 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506644+0000", 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6169999999999999e-06 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506716+0000", 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2272000000000002e-05 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506731+0000", 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5065999999999999e-05 2024-09-16T03:41:16.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506739+0000", 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2230000000000007e-06 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.508774+0000", 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002034264 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.508867+0000", 2024-09-16T03:41:16.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3713000000000002e-05 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.509403+0000", 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00053514299999999997 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.512521+0000", 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031181920000000001 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:16.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.533480+0000", 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020959177999999998 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.533563+0000", 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3331000000000006e-05 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:894 3.1s0 3:82a78c8d:::smithi12138290-1:head [setxattr _header (56) in=7b] snapc 1d=[1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.506763+0000", 2024-09-16T03:41:16.855 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.296399629, 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.028329701999999998, 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 894 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506763+0000", 2024-09-16T03:41:16.856 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506763+0000", 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506764+0000", 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2999999999999999e-07 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506770+0000", 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2219999999999999e-06 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506778+0000", 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4610000000000002e-06 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506785+0000", 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0790000000000003e-06 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.510080+0000", 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032957709999999999 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.510184+0000", 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000103644 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.510640+0000", 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000455823 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.515070+0000", 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044299789999999997 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.535029+0000", 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.019959147999999999 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.535093+0000", 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3744999999999994e-05 2024-09-16T03:41:16.860 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:895 3.1s0 3:82a78c8d:::smithi12138290-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.506796+0000", 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.296367010000001, 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.029474764000000001, 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.861 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 895 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506796+0000", 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506796+0000", 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.862 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506796+0000", 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5599999999999995e-07 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506799+0000", 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.497e-06 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506808+0000", 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6379999999999999e-06 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.863 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506815+0000", 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3130000000000004e-06 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.510904+0000", 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040891809999999999 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.510973+0000", 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8979999999999993e-05 2024-09-16T03:41:16.864 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.535419+0000", 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.024446045999999999 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.535479+0000", 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0438e-05 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.536270+0000", 2024-09-16T03:41:16.865 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000791115 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:970 3.1s0 3:8a735ecd:::smithi12138290-30:head [copy-from ver 34] snapc 24=[24,23,22,21,20,1f,1e,1d,1b,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:11.494650+0000", 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.3085127679999999, 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020428584999999999, 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.866 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 970 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494650+0000", 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494650+0000", 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.867 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494650+0000", 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0800000000000001e-07 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494655+0000", 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7310000000000001e-06 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494659+0000", 2024-09-16T03:41:16.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.754e-06 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494661+0000", 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.661e-06 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.498097+0000", 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00343583 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.498182+0000", 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4753000000000007e-05 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.506976+0000", 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087937889999999998 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.509015+0000", 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020394739999999999 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.515026+0000", 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060104900000000003 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.515078+0000", 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2794999999999999e-05 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:971 3.1s0 3:8a735ecd:::smithi12138290-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:41:16.871 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:11.494665+0000", 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.3084973599999996, 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014207153, 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 971 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.872 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494665+0000", 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494665+0000", 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494666+0000", 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4999999999999999e-07 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494667+0000", 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.381e-06 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494670+0000", 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3419999999999999e-06 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494674+0000", 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4489999999999999e-06 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.498334+0000", 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036606439999999998 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.498378+0000", 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3269000000000002e-05 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.508498+0000", 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010119981 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.875 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.508607+0000", 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010894400000000001 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.508872+0000", 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026589300000000001 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 94/55 ECSubWrite(tid=206, reqid=osd.1.0:206, at_version=94'90, trim_to=0'0, roll_forward_to=91'88))", 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.482149+0000", 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.321014028, 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017282457000000001, 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482149+0000", 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482149+0000", 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482149+0000", 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482177+0000", 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8127e-05 2024-09-16T03:41:16.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482188+0000", 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0482000000000001e-05 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482197+0000", 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.816e-06 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483361+0000", 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001164872 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483373+0000", 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1331999999999999e-05 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.499395+0000", 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016022674000000001 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.499431+0000", 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5593999999999998e-05 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s1 94/31 ECSubWrite(tid=101, reqid=osd.8.0:101, at_version=94'79, trim_to=0'0, roll_forward_to=91'75))", 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.482616+0000", 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.3205470720000001, 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011972284999999999, 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.881 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482616+0000", 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482616+0000", 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.882 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482616+0000", 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9299999999999998e-07 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482639+0000", 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2908000000000002e-05 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482652+0000", 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2656999999999999e-05 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482661+0000", 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9639999999999992e-06 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482724+0000", 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2813999999999999e-05 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482734+0000", 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0282000000000001e-05 2024-09-16T03:41:16.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.494418+0000", 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011683717 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.494588+0000", 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017034999999999999 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 94/55 ECSubWrite(tid=208, reqid=osd.1.0:208, at_version=94'92, trim_to=0'0, roll_forward_to=91'88))", 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.483746+0000", 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.3194170079999998, 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015709785, 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483746+0000", 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.886 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.887 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.887 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.887 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483746+0000", 2024-09-16T03:41:16.887 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.887 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.887 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.887 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.887 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483747+0000", 2024-09-16T03:41:16.887 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9000000000000005e-07 2024-09-16T03:41:16.887 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483772+0000", 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5097999999999999e-05 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483785+0000", 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3010000000000001e-05 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.888 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483794+0000", 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9309999999999997e-06 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485884+0000", 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020905559999999999 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485903+0000", 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9074000000000001e-05 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.890 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.890 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.499435+0000", 2024-09-16T03:41:16.890 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013532089000000001 2024-09-16T03:41:16.890 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.890 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.890 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.890 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.499455+0000", 2024-09-16T03:41:16.890 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0037000000000001e-05 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds6 94/85 ECSubWrite(tid=212, reqid=osd.14.0:212, at_version=94'108, trim_to=0'0, roll_forward_to=91'104))", 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.485800+0000", 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.3173628719999999, 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012674903, 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.891 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485800+0000", 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485800+0000", 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.892 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485801+0000", 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9800000000000002e-07 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485826+0000", 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4898000000000001e-05 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485838+0000", 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2523e-05 2024-09-16T03:41:16.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485860+0000", 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2384000000000001e-05 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.489427+0000", 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0035664410000000001 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.489442+0000", 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4868000000000001e-05 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.498444+0000", 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090022910000000008 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.498475+0000", 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0599999999999998e-05 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.895 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 97/55 ECSubWrite(tid=219, reqid=osd.1.0:219, at_version=97'98, trim_to=0'0, roll_forward_to=95'96))", 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.481602+0000", 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.321561021, 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013086326000000001, 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.896 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.481602+0000", 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.481602+0000", 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.481602+0000", 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0100000000000001e-07 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.897 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.481614+0000", 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2279000000000001e-05 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482579+0000", 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00096509299999999998 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482590+0000", 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0523000000000001e-05 2024-09-16T03:41:16.898 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482724+0000", 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013399500000000001 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482733+0000", 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1929999999999999e-06 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.494661+0000", 2024-09-16T03:41:16.899 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011928533999999999 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.494688+0000", 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6508000000000001e-05 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 97/55 ECSubWrite(tid=221, reqid=client.4615.0:995, at_version=97'100, trim_to=0'0, roll_forward_to=95'96))", 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.484613+0000", 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.31854946000000001, 2024-09-16T03:41:16.900 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017659327999999998, 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484613+0000", 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484613+0000", 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484614+0000", 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0999999999999998e-07 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484633+0000", 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9366999999999999e-05 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.902 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.903 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484652+0000", 2024-09-16T03:41:16.903 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9177999999999998e-05 2024-09-16T03:41:16.903 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.903 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.903 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.903 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484660+0000", 2024-09-16T03:41:16.903 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7729999999999999e-06 2024-09-16T03:41:16.903 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.486785+0000", 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002125002 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.486795+0000", 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9450000000000005e-06 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.904 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502244+0000", 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015448638000000001 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502273+0000", 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8714999999999999e-05 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 97/55 ECSubWrite(tid=222, reqid=client.4615.0:996, at_version=97'101, trim_to=0'0, roll_forward_to=95'96))", 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.484683+0000", 2024-09-16T03:41:16.905 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.31847964200000001, 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017620747999999999, 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484683+0000", 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.906 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484683+0000", 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484683+0000", 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0699999999999998e-07 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484698+0000", 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4990999999999999e-05 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.907 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484706+0000", 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7020000000000001e-06 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484712+0000", 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3749999999999999e-06 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.491681+0000", 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006968602 2024-09-16T03:41:16.908 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.491689+0000", 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502278+0000", 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010589215000000001 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.909 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502304+0000", 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5372999999999999e-05 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 97/55 ECSubWrite(tid=223, reqid=client.4615.0:997, at_version=97'102, trim_to=0'0, roll_forward_to=95'96))", 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.487124+0000", 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.31603900200000001, 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015202571, 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:16.910 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487124+0000", 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487124+0000", 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487124+0000", 2024-09-16T03:41:16.911 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.51e-07 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487141+0000", 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.717e-05 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487151+0000", 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.012e-05 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:16.912 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487160+0000", 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0539999999999995e-06 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.496120+0000", 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0089605789999999998 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.496130+0000", 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4089999999999992e-06 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.913 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502306+0000", 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061766759999999999 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502326+0000", 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0012000000000001e-05 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:16.914 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:16.916 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:16.913+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:16.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:41:17.003 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.12 2024-09-16T03:41:17.004 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-12 fsck 2024-09-16T03:41:17.016 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:17.014+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:17.018 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:17.018 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:17.018 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:41:17.018 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:17.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:41:17.117 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:17.116+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:17.170 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:41:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:41:17.171 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:17.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:41:17.218 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:17.217+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:17.283 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:17.283 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:41:17.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:41:17.283 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:41:17.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:282 3.4s0 3:2307c0f5:::smithi12138290-28:head [append 0~458752] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 54.705743435000002, 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0093867299999999994, 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 282 2024-09-16T03:41:17.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574793+0000", 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5799999999999999e-06 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575536+0000", 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00074309099999999998 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575567+0000", 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0382000000000002e-05 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575590+0000", 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3215000000000001e-05 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575665+0000", 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5462000000000004e-05 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575671+0000", 2024-09-16T03:41:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6969999999999998e-06 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577693+0000", 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020222439999999999 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577796+0000", 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010306000000000001 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578998+0000", 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012013880000000001 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.583553+0000", 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045549340000000001 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584072+0000", 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00051922199999999996 2024-09-16T03:41:17.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584177+0000", 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105455 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:311 3.bs0 3:db21b75b:::smithi12138290-3:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 48.789591332999997, 2024-09-16T03:41:17.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013432349, 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 311 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:41:17.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:41:17.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490945+0000", 2024-09-16T03:41:17.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.419e-06 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490953+0000", 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7379999999999994e-06 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490973+0000", 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0460999999999999e-05 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490984+0000", 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0857e-05 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491053+0000", 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8351999999999999e-05 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491061+0000", 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3569999999999998e-06 2024-09-16T03:41:17.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493588+0000", 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025267430000000001 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493683+0000", 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4876000000000004e-05 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496393+0000", 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027106500000000002 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504203+0000", 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078100080000000002 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504276+0000", 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2601000000000005e-05 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.296 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504375+0000", 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9287e-05 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:312 3.bs0 3:db21b75b:::smithi12138290-3:head [rollback a] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 48.789525707999999, 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013925697000000001, 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.297 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 312 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491009+0000", 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0200000000000001e-07 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491013+0000", 2024-09-16T03:41:17.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8160000000000004e-06 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491022+0000", 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6649999999999992e-06 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491028+0000", 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4760000000000003e-06 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491124+0000", 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6192000000000003e-05 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491132+0000", 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6539999999999999e-06 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499113+0000", 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079811880000000002 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499184+0000", 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0749999999999999e-05 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499557+0000", 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037325200000000002 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504380+0000", 2024-09-16T03:41:17.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048224590000000003 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504852+0000", 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047194899999999999 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504934+0000", 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2594000000000002e-05 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:313 3.bs0 3:db21b75b:::smithi12138290-3:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 48.789484066999997, 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016369406999999999, 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.304 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 313 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491051+0000", 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.82e-07 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491054+0000", 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5260000000000002e-06 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.306 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491063+0000", 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.354e-06 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491069+0000", 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2299999999999996e-06 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491177+0000", 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000108285 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491184+0000", 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0489999999999998e-06 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501958+0000", 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010773448 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502064+0000", 2024-09-16T03:41:17.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106661 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502545+0000", 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000481077 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504381+0000", 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018352970000000001 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.507349+0000", 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002968666 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.507420+0000", 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0232000000000001e-05 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:369 3.4s0 3:2ef36fe6:::smithi12138290-31:head [append 0~131072] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:41:17.310 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 44.750953580999997, 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090412610000000001, 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 369 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.311 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529581+0000", 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2399999999999998e-07 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529661+0000", 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9846999999999996e-05 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529666+0000", 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1919999999999996e-06 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529670+0000", 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3830000000000001e-06 2024-09-16T03:41:17.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.530925+0000", 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012551730000000001 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.531047+0000", 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121934 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.531824+0000", 2024-09-16T03:41:17.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000777384 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.534187+0000", 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023626609999999998 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538565+0000", 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043779170000000003 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538622+0000", 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7246e-05 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:370 3.4s0 3:2ef36fe6:::smithi12138290-31:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.529676+0000", 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 44.750857777, 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092009889999999997, 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.316 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 370 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529676+0000", 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529676+0000", 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529677+0000", 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9099999999999999e-07 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529679+0000", 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7799999999999999e-06 2024-09-16T03:41:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529682+0000", 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4209999999999999e-06 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529685+0000", 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3459999999999998e-06 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533114+0000", 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034284559999999999 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533215+0000", 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010074200000000001 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533660+0000", 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044525100000000002 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.320 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.534705+0000", 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001044971 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538781+0000", 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040766639999999998 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.538877+0000", 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5966999999999998e-05 2024-09-16T03:41:17.321 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:371 3.4s0 3:2ef36fe6:::smithi12138290-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 44.750844995999998, 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0095505, 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.322 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 371 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:41:17.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7799999999999997e-07 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529691+0000", 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2330000000000001e-06 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529694+0000", 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5489999999999998e-06 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529697+0000", 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.706e-06 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533902+0000", 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004204564 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533982+0000", 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0399000000000001e-05 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.539228+0000", 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052458579999999999 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.539240+0000", 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1912999999999999e-05 2024-09-16T03:41:17.326 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(osd.11.23:1 3.bs0 3:de0fee80:::smithi12138290-48:head [assert-version v28,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.479910+0000", 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 34.800623707, 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092349349999999997, 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479910+0000", 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479910+0000", 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479913+0000", 2024-09-16T03:41:17.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0600000000000002e-06 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479920+0000", 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6290000000000001e-06 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479936+0000", 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5749000000000001e-05 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479945+0000", 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4329999999999993e-06 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480018+0000", 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2384000000000004e-05 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480125+0000", 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107074 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.489145+0000", 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090206060000000005 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 73/31 ECSubWrite(tid=176, reqid=osd.0.0:176, at_version=73'82, trim_to=0'0, roll_forward_to=72'78))", 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.481052+0000", 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.799482017999999, 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012524485, 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.331 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481052+0000", 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481052+0000", 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481053+0000", 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9300000000000006e-07 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481077+0000", 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4111999999999999e-05 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481098+0000", 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0704e-05 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481107+0000", 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8820000000000007e-06 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481195+0000", 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7662000000000002e-05 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481204+0000", 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3980000000000005e-06 2024-09-16T03:41:17.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493544+0000", 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012339902 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493577+0000", 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2832e-05 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:649 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~425984] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.483387+0000", 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.797147016, 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.021278157999999998, 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 649 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.336 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.483387+0000", 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.483387+0000", 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.483389+0000", 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1600000000000001e-06 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.484029+0000", 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00063944299999999996 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.484041+0000", 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2369e-05 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.484049+0000", 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9619999999999993e-06 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.486874+0000", 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028244390000000002 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487037+0000", 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016383700000000001 2024-09-16T03:41:17.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.488288+0000", 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001250995 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493564+0000", 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052756249999999999 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504598+0000", 2024-09-16T03:41:17.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011034089 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504665+0000", 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7238999999999998e-05 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:650 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~606208] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.487370+0000", 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.793164675, 2024-09-16T03:41:17.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017507029, 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 650 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487370+0000", 2024-09-16T03:41:17.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487370+0000", 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487371+0000", 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9570000000000001e-06 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.343 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487965+0000", 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00059316399999999997 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487986+0000", 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1182000000000001e-05 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487996+0000", 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0146000000000001e-05 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.490298+0000", 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023018779999999998 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.490410+0000", 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000112457 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492647+0000", 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002236465 2024-09-16T03:41:17.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499318+0000", 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066710809999999997 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504802+0000", 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054842770000000001 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504877+0000", 2024-09-16T03:41:17.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4422e-05 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:651 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~540672] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.491437+0000", 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.789097654999999, 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013890236, 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.347 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 651 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491437+0000", 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491437+0000", 2024-09-16T03:41:17.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491438+0000", 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.578e-06 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492172+0000", 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073402799999999996 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.349 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492186+0000", 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4137e-05 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492195+0000", 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2619999999999998e-06 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493674+0000", 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001479803 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493779+0000", 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000104399 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.494705+0000", 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00092621299999999995 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499319+0000", 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046143490000000002 2024-09-16T03:41:17.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505255+0000", 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059355179999999999 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505327+0000", 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1948999999999998e-05 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:652 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~409600] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.495027+0000", 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.785506849000001, 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010815300999999999, 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 652 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495027+0000", 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495027+0000", 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495029+0000", 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0789999999999999e-06 2024-09-16T03:41:17.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495441+0000", 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000411753 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495457+0000", 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495466+0000", 2024-09-16T03:41:17.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9279999999999999e-06 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495851+0000", 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038526000000000001 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496004+0000", 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015289200000000001 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496867+0000", 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00086269600000000004 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504685+0000", 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078182930000000005 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505774+0000", 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010881599999999999 2024-09-16T03:41:17.357 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505843+0000", 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9101000000000001e-05 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:653 3.bs0 3:de0fee80:::smithi12138290-48:head [setxattr _header (57) in=7b] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.785051648, 2024-09-16T03:41:17.358 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013098884, 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 653 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.359 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4600000000000003e-07 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495487+0000", 2024-09-16T03:41:17.360 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9389999999999999e-06 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495495+0000", 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4100000000000002e-06 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495500+0000", 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6300000000000003e-06 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.361 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499810+0000", 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043094860000000004 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499920+0000", 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000110507 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.500346+0000", 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000425735 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.362 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504687+0000", 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043411109999999999 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508462+0000", 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037750840000000002 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508581+0000", 2024-09-16T03:41:17.363 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011913600000000001 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:654 3.bs0 3:de0fee80:::smithi12138290-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.785024591999999, 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016989471999999999, 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.364 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 654 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.365 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3900000000000005e-07 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495512+0000", 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2259999999999999e-06 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495519+0000", 2024-09-16T03:41:17.366 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7769999999999997e-06 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495525+0000", 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4419999999999997e-06 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.500653+0000", 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051282619999999998 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.500758+0000", 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105141 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508620+0000", 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078621590000000005 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508703+0000", 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2622999999999998e-05 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.512499+0000", 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003796303 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 74/31 ECSubWrite(tid=194, reqid=osd.0.0:194, at_version=74'100, trim_to=0'0, roll_forward_to=73'96))", 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.530401+0000", 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 23.750132851, 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092564819999999999, 2024-09-16T03:41:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530401+0000", 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530401+0000", 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530402+0000", 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2900000000000004e-07 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530430+0000", 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7979000000000001e-05 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530441+0000", 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0855e-05 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530449+0000", 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1750000000000005e-06 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530878+0000", 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00042890400000000003 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530941+0000", 2024-09-16T03:41:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3035999999999998e-05 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539613+0000", 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086722910000000004 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539658+0000", 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4712999999999997e-05 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:790 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~0] snapc 1b=[1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:57.545852+0000", 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.734682121999999, 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0097149109999999997, 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 790 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545852+0000", 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545852+0000", 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545852+0000", 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.32e-07 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545857+0000", 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4410000000000003e-06 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545863+0000", 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2729999999999999e-06 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545869+0000", 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3739999999999997e-06 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.548747+0000", 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028787299999999999 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.548802+0000", 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4846000000000003e-05 2024-09-16T03:41:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.549282+0000", 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047939 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.552414+0000", 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031319659999999999 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.555487+0000", 2024-09-16T03:41:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030735099999999998 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.555567+0000", 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9949000000000003e-05 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s4 84/31 ECSubWrite(tid=89, reqid=osd.8.0:89, at_version=84'71, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.485270+0000", 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.795264261, 2024-09-16T03:41:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010818193, 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485270+0000", 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485270+0000", 2024-09-16T03:41:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485271+0000", 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000001e-07 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485288+0000", 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7501000000000001e-05 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485304+0000", 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6279000000000001e-05 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485313+0000", 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1920000000000005e-06 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485372+0000", 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9685000000000002e-05 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485382+0000", 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4539999999999996e-06 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496052+0000", 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010669931000000001 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496088+0000", 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6473000000000001e-05 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.8s1 84/31 ECSubWrite(tid=220, reqid=osd.11.0:220, at_version=84'33, trim_to=0'0, roll_forward_to=83'31))", 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.489139+0000", 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 13.791395640999999, 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0097139679999999999, 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489139+0000", 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489139+0000", 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489139+0000", 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2200000000000003e-07 2024-09-16T03:41:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489162+0000", 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3059999999999999e-05 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489177+0000", 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4408e-05 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489185+0000", 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5969999999999992e-06 2024-09-16T03:41:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489252+0000", 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6661000000000004e-05 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489262+0000", 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0373999999999999e-05 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.498811+0000", 2024-09-16T03:41:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.009548974 2024-09-16T03:41:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.498853+0000", 2024-09-16T03:41:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1171999999999998e-05 2024-09-16T03:41:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.389 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.389 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.389 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.389 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.389 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 97/88 ECSubWrite(tid=408, reqid=osd.0.0:408, at_version=97'213, trim_to=0'0, roll_forward_to=95'209))", 2024-09-16T03:41:17.389 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.490360+0000", 2024-09-16T03:41:17.389 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.79017388600000005, 2024-09-16T03:41:17.389 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010242849, 2024-09-16T03:41:17.389 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.389 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.390 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:17.390 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.390 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.390 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.390 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490360+0000", 2024-09-16T03:41:17.390 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.390 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.391 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.391 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.391 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490360+0000", 2024-09-16T03:41:17.391 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.391 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.391 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.391 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.392 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490361+0000", 2024-09-16T03:41:17.392 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5199999999999996e-07 2024-09-16T03:41:17.392 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.392 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.392 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.392 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490380+0000", 2024-09-16T03:41:17.393 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8499999999999999e-05 2024-09-16T03:41:17.393 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.393 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.393 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.393 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490394+0000", 2024-09-16T03:41:17.393 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4338999999999999e-05 2024-09-16T03:41:17.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.394 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.394 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.394 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490449+0000", 2024-09-16T03:41:17.394 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4824999999999998e-05 2024-09-16T03:41:17.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.394 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.395 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.395 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490733+0000", 2024-09-16T03:41:17.395 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00028452600000000001 2024-09-16T03:41:17.395 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.395 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.395 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.396 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490742+0000", 2024-09-16T03:41:17.396 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.867e-06 2024-09-16T03:41:17.396 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.396 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.396 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.396 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.500570+0000", 2024-09-16T03:41:17.397 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0098277280000000009 2024-09-16T03:41:17.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.397 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.397 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.397 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.500603+0000", 2024-09-16T03:41:17.397 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3312000000000002e-05 2024-09-16T03:41:17.398 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.398 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.398 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.398 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.398 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.398 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:17.401 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:41:17.402 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:41:17.398+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:41:17.479 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T03:41:17.482 INFO:tasks.rados.rados.0.smithi121.stdout:380: done (2 left) 2024-09-16T03:41:17.482 INFO:tasks.rados.rados.0.smithi121.stdout:381: rmattr oid 46 current snap is 37 2024-09-16T03:41:17.483 INFO:tasks.rados.rados.0.smithi121.stdout:381: done (2 left) 2024-09-16T03:41:17.483 INFO:tasks.rados.rados.0.smithi121.stdout:382: read oid 36 snap -1 2024-09-16T03:41:17.483 INFO:tasks.rados.rados.0.smithi121.stdout:382: expect (ObjNum 112 snap 23 seq_num 112) 2024-09-16T03:41:17.483 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 28 current snap is 37 2024-09-16T03:41:17.483 INFO:tasks.rados.rados.0.smithi121.stdout:383: seq_num 161 ranges {1687552=311296} 2024-09-16T03:41:17.494 INFO:tasks.rados.rados.0.smithi121.stdout:383: writing smithi12138290-28 from 1687552 to 1998848 tid 1 2024-09-16T03:41:17.494 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 5 current snap is 37 2024-09-16T03:41:17.494 INFO:tasks.rados.rados.0.smithi121.stdout:379: finishing rollback tid 0 to smithi12138290-18 2024-09-16T03:41:17.494 INFO:tasks.rados.rados.0.smithi121.stdout:379: finishing rollback tid 1 to smithi12138290-18 2024-09-16T03:41:17.495 INFO:tasks.rados.rados.0.smithi121.stdout:378: finishing rollback tid 0 to smithi12138290-22 2024-09-16T03:41:17.495 INFO:tasks.rados.rados.0.smithi121.stdout:378: finishing rollback tid 1 to smithi12138290-22 2024-09-16T03:41:17.495 INFO:tasks.rados.rados.0.smithi121.stdout:378: finishing rollback tid 2 to smithi12138290-22 2024-09-16T03:41:17.495 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 22 v 102 (ObjNum 152 snap 31 seq_num 152) dirty exists 2024-09-16T03:41:17.495 INFO:tasks.rados.rados.0.smithi121.stdout:379: finishing rollback tid 2 to smithi12138290-18 2024-09-16T03:41:17.495 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 18 v 189 (ObjNum 89 snap 12 seq_num 89) dirty exists 2024-09-16T03:41:17.495 INFO:tasks.rados.rados.0.smithi121.stdout:384: seq_num 162 ranges {3866624=327680} 2024-09-16T03:41:17.501 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:17.499+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:17.515 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:17.515 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:17.515 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:41:17.515 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:17.519 INFO:tasks.rados.rados.0.smithi121.stdout:384: writing smithi12138290-5 from 3866624 to 4194304 tid 1 2024-09-16T03:41:17.519 INFO:tasks.rados.rados.0.smithi121.stdout:378: done (4 left) 2024-09-16T03:41:17.519 INFO:tasks.rados.rados.0.smithi121.stdout:379: done (3 left) 2024-09-16T03:41:17.519 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 8 current snap is 37 2024-09-16T03:41:17.519 INFO:tasks.rados.rados.0.smithi121.stdout:385: seq_num 163 ranges {131072=344064} 2024-09-16T03:41:17.522 INFO:tasks.rados.rados.0.smithi121.stdout:385: writing smithi12138290-8 from 131072 to 475136 tid 1 2024-09-16T03:41:17.522 INFO:tasks.rados.rados.0.smithi121.stdout:386: setattr oid 43 current snap is 37 2024-09-16T03:41:17.522 INFO:tasks.rados.rados.0.smithi121.stdout:382: expect (ObjNum 112 snap 23 seq_num 112) 2024-09-16T03:41:17.526 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:41:17.572 INFO:tasks.rados.rados.0.smithi121.stdout:383: finishing write tid 1 to smithi12138290-28 2024-09-16T03:41:17.572 INFO:tasks.rados.rados.0.smithi121.stdout:383: finishing write tid 2 to smithi12138290-28 2024-09-16T03:41:17.573 INFO:tasks.rados.rados.0.smithi121.stdout:383: finishing write tid 3 to smithi12138290-28 2024-09-16T03:41:17.573 INFO:tasks.rados.rados.0.smithi121.stdout:383: oid 28 updating version 0 to 47 2024-09-16T03:41:17.573 INFO:tasks.rados.rados.0.smithi121.stdout:383: oid 28 version 47 is already newer than 46 2024-09-16T03:41:17.573 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 28 v 47 (ObjNum 161 snap 37 seq_num 161) dirty exists 2024-09-16T03:41:17.573 INFO:tasks.rados.rados.0.smithi121.stdout:383: left oid 28 (ObjNum 161 snap 37 seq_num 161) 2024-09-16T03:41:17.573 INFO:tasks.rados.rados.0.smithi121.stdout:384: finishing write tid 1 to smithi12138290-5 2024-09-16T03:41:17.573 INFO:tasks.rados.rados.0.smithi121.stdout:384: finishing write tid 2 to smithi12138290-5 2024-09-16T03:41:17.573 INFO:tasks.rados.rados.0.smithi121.stdout:385: finishing write tid 1 to smithi12138290-8 2024-09-16T03:41:17.573 INFO:tasks.rados.rados.0.smithi121.stdout:385: finishing write tid 2 to smithi12138290-8 2024-09-16T03:41:17.573 INFO:tasks.rados.rados.0.smithi121.stdout:384: finishing write tid 3 to smithi12138290-5 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:384: oid 5 updating version 0 to 75 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:384: oid 5 updating version 75 to 76 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 5 v 76 (ObjNum 162 snap 37 seq_num 162) dirty exists 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:384: left oid 5 (ObjNum 162 snap 37 seq_num 162) 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:385: finishing write tid 3 to smithi12138290-8 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:385: oid 8 updating version 0 to 201 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:385: oid 8 updating version 201 to 202 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 8 v 202 (ObjNum 163 snap 37 seq_num 163) dirty exists 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:385: left oid 8 (ObjNum 163 snap 37 seq_num 163) 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:382: done (4 left) 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:383: done (3 left) 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:384: done (2 left) 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:385: done (1 left) 2024-09-16T03:41:17.574 INFO:tasks.rados.rados.0.smithi121.stdout:387: setattr oid 39 current snap is 37 2024-09-16T03:41:17.575 INFO:tasks.rados.rados.0.smithi121.stdout:388: copy_from oid 2 from oid 17 current snap is 37 2024-09-16T03:41:17.575 INFO:tasks.rados.rados.0.smithi121.stdout:389: snap_create 2024-09-16T03:41:17.578 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 39 v 109 (ObjNum 146 snap 29 seq_num 146) dirty exists 2024-09-16T03:41:17.580 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 43 v 227 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:41:17.588 INFO:tasks.rados.rados.0.smithi121.stdout:388: finishing copy_from racing read to smithi12138290-2 2024-09-16T03:41:17.594 INFO:tasks.rados.rados.0.smithi121.stdout:388: finishing copy_from to smithi12138290-2 2024-09-16T03:41:17.594 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 2 v 166 (ObjNum 118 snap 24 seq_num 118) dirty exists 2024-09-16T03:41:17.602 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:17.601+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:17.633 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:17.634 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:17.634 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:41:17.634 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:41:17.634 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:17.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:41:17.703 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:41:17.701+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:41:17.747 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:17.747 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:41:17.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:41:17.747 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:41:17.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.747 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:205 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~655360] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 61.221189565000003, 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075538640000000004, 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 205 2024-09-16T03:41:17.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523025+0000", 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3519999999999999e-06 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523819+0000", 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00079422800000000001 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523834+0000", 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5333e-05 2024-09-16T03:41:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523842+0000", 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2960000000000001e-06 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523903+0000", 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1124000000000006e-05 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523981+0000", 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8166999999999999e-05 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525035+0000", 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010540969999999999 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528679+0000", 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036443090000000001 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530521+0000", 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018416789999999999 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530577+0000", 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6279000000000001e-05 2024-09-16T03:41:17.753 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:305 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:41:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 49.256963831, 2024-09-16T03:41:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087742589999999995, 2024-09-16T03:41:17.754 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 305 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:41:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487251+0000", 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.824e-06 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487259+0000", 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1689999999999992e-06 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487275+0000", 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5889e-05 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487283+0000", 2024-09-16T03:41:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9710000000000004e-06 2024-09-16T03:41:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487385+0000", 2024-09-16T03:41:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010225099999999999 2024-09-16T03:41:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487554+0000", 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016910199999999999 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489000+0000", 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001445499 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492361+0000", 2024-09-16T03:41:17.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033615799999999999 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495935+0000", 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0035732899999999998 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496023+0000", 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8684000000000007e-05 2024-09-16T03:41:17.760 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.761 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:306 3.6s0 3:64535dd7:::smithi12138290-18:head [rollback 5] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:41:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 49.256910969000003, 2024-09-16T03:41:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0099232220000000006, 2024-09-16T03:41:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.762 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 306 2024-09-16T03:41:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:41:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:41:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487303+0000", 2024-09-16T03:41:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.44e-07 2024-09-16T03:41:17.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487306+0000", 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.817e-06 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487314+0000", 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4000000000000003e-06 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487319+0000", 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2379999999999997e-06 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489535+0000", 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002216347 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489617+0000", 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1559999999999998e-05 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490002+0000", 2024-09-16T03:41:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038484000000000001 2024-09-16T03:41:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492363+0000", 2024-09-16T03:41:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023609070000000002 2024-09-16T03:41:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497154+0000", 2024-09-16T03:41:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047910110000000004 2024-09-16T03:41:17.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497225+0000", 2024-09-16T03:41:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1557999999999995e-05 2024-09-16T03:41:17.774 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:307 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:41:17.775 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 49.256869152999997, 2024-09-16T03:41:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010112101, 2024-09-16T03:41:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.776 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.777 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 307 2024-09-16T03:41:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:41:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:41:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:41:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.51e-07 2024-09-16T03:41:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487356+0000", 2024-09-16T03:41:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1793999999999999e-05 2024-09-16T03:41:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487362+0000", 2024-09-16T03:41:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8100000000000003e-06 2024-09-16T03:41:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487366+0000", 2024-09-16T03:41:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8670000000000004e-06 2024-09-16T03:41:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490302+0000", 2024-09-16T03:41:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029359299999999998 2024-09-16T03:41:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490396+0000", 2024-09-16T03:41:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4134000000000004e-05 2024-09-16T03:41:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490929+0000", 2024-09-16T03:41:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00053311500000000002 2024-09-16T03:41:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492364+0000", 2024-09-16T03:41:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014347279999999999 2024-09-16T03:41:17.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497383+0000", 2024-09-16T03:41:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050189220000000003 2024-09-16T03:41:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497456+0000", 2024-09-16T03:41:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3250000000000005e-05 2024-09-16T03:41:17.790 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.791 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:349 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [rollback 3] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:41:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 45.245914616999997, 2024-09-16T03:41:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084181240000000008, 2024-09-16T03:41:17.792 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.793 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 349 2024-09-16T03:41:17.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:41:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:41:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498299+0000", 2024-09-16T03:41:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6600000000000002e-07 2024-09-16T03:41:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498301+0000", 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3860000000000001e-06 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498307+0000", 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8590000000000001e-06 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498312+0000", 2024-09-16T03:41:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6249999999999998e-06 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499617+0000", 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0013049660000000001 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499672+0000", 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5787e-05 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500118+0000", 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044587499999999998 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505378+0000", 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005259789 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506634+0000", 2024-09-16T03:41:17.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012560169999999999 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506716+0000", 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2354000000000005e-05 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:350 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:17.802 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:41:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 45.245894544999999, 2024-09-16T03:41:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086865829999999995, 2024-09-16T03:41:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.803 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 350 2024-09-16T03:41:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:41:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:41:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498319+0000", 2024-09-16T03:41:17.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.39e-07 2024-09-16T03:41:17.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498321+0000", 2024-09-16T03:41:17.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.815e-06 2024-09-16T03:41:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498325+0000", 2024-09-16T03:41:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8749999999999999e-06 2024-09-16T03:41:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498329+0000", 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.014e-06 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500398+0000", 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020690069999999999 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500458+0000", 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9429999999999999e-05 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500725+0000", 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026708899999999998 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505380+0000", 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046551040000000002 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506933+0000", 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015530769999999999 2024-09-16T03:41:17.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507005+0000", 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1833000000000002e-05 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s2 53/31 ECSubWrite(tid=40, reqid=osd.13.0:40, at_version=53'20, trim_to=0'0, roll_forward_to=52'18))", 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 45.242627212999999, 2024-09-16T03:41:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007493184, 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:41:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.23e-07 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501604+0000", 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8465e-05 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501615+0000", 2024-09-16T03:41:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.06e-05 2024-09-16T03:41:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501623+0000", 2024-09-16T03:41:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8019999999999992e-06 2024-09-16T03:41:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501779+0000", 2024-09-16T03:41:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000156093 2024-09-16T03:41:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501788+0000", 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3840000000000003e-06 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.509033+0000", 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0072446300000000002 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.509079+0000", 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5986999999999999e-05 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.818 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:412 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~737280] snapc d=[d,b,8,5,3] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 45.141281806000002, 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007659028, 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.819 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 412 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602932+0000", 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3499999999999995e-07 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603668+0000", 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073570899999999997 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603677+0000", 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4180000000000003e-06 2024-09-16T03:41:17.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603702+0000", 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5225999999999999e-05 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603754+0000", 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1391000000000003e-05 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603826+0000", 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2648e-05 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604587+0000", 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076084500000000001 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.608450+0000", 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038632409999999999 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610532+0000", 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020811580000000001 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610590+0000", 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8656999999999998e-05 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.824 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:466 3.6s0 3:6167ac73:::smithi12138290-7:head [stat,delete] snapc 11=[11,10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 39.269168458999999, 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0074852260000000002, 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 466 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475046+0000", 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3349999999999999e-06 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475057+0000", 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1158999999999999e-05 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475069+0000", 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1640999999999999e-05 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475076+0000", 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5379999999999997e-06 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475130+0000", 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4267000000000001e-05 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:17.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475135+0000", 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8640000000000004e-06 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478234+0000", 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030989450000000001 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478358+0000", 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000123666 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478978+0000", 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00061992000000000004 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.479700+0000", 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00072176900000000003 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.482485+0000", 2024-09-16T03:41:17.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027855340000000001 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.482530+0000", 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4588000000000001e-05 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:688 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~475136] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.539837+0000", 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.204376344, 2024-09-16T03:41:17.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010608545, 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 688 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539837+0000", 2024-09-16T03:41:17.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539837+0000", 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539839+0000", 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2479999999999999e-06 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540778+0000", 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00093899100000000002 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540797+0000", 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9191e-05 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540806+0000", 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6200000000000005e-06 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540874+0000", 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7949999999999998e-05 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.541002+0000", 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000128131 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542267+0000", 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012655019999999999 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550290+0000", 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080230189999999993 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550352+0000", 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2162999999999994e-05 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550445+0000", 2024-09-16T03:41:17.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2730000000000002e-05 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:689 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~655360] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.544092+0000", 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 24.200120811000001, 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081954690000000004, 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.837 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 689 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544092+0000", 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544092+0000", 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544094+0000", 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8619999999999999e-06 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545122+0000", 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010280389999999999 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545135+0000", 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3426e-05 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545143+0000", 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8269999999999998e-06 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545656+0000", 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00051290899999999998 2024-09-16T03:41:17.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545817+0000", 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016028200000000001 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.546825+0000", 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001008253 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550450+0000", 2024-09-16T03:41:17.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036248209999999999 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.552236+0000", 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017865380000000001 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.552288+0000", 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1511999999999997e-05 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:821 3.6s0 3:616ac6f8:::smithi12138290-8:head [append 0~0] snapc 1c=[1c,1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:58.596286+0000", 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.147927342999999, 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080138029999999999, 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 821 2024-09-16T03:41:17.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596286+0000", 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596286+0000", 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596287+0000", 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.699e-06 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596298+0000", 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1104000000000001e-05 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596316+0000", 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7652999999999999e-05 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596324+0000", 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0690000000000002e-06 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596454+0000", 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000129863 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596460+0000", 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9540000000000001e-06 2024-09-16T03:41:17.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.598917+0000", 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024574169999999999 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.599031+0000", 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011393800000000001 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.599896+0000", 2024-09-16T03:41:17.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000864319 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.602014+0000", 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021184149999999998 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.604224+0000", 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022102519999999998 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.604299+0000", 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5119999999999999e-05 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:822 3.6s0 3:616ac6f8:::smithi12138290-8:head [rollback 11] snapc 1c=[1c,1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:58.596341+0000", 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.147872185000001, 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011007497999999999, 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.849 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 822 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596341+0000", 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596341+0000", 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596341+0000", 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6199999999999999e-07 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596343+0000", 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9e-06 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596348+0000", 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596351+0000", 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.202e-06 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596536+0000", 2024-09-16T03:41:17.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000184926 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596541+0000", 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.318e-06 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.600433+0000", 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038927150000000002 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.600496+0000", 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.600857+0000", 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00036071600000000002 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.602040+0000", 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001182673 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.607301+0000", 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052608639999999996 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.607348+0000", 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7763000000000003e-05 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.855 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:823 3.6s0 3:616ac6f8:::smithi12138290-8:head [append 0~0] snapc 1c=[1c,1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:58.596421+0000", 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.147792179, 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011103531999999999, 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 823 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.856 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596421+0000", 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596421+0000", 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596421+0000", 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.84e-07 2024-09-16T03:41:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596423+0000", 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0490000000000002e-06 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596428+0000", 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3129999999999998e-06 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596432+0000", 2024-09-16T03:41:17.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.196e-06 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596569+0000", 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000137121 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596572+0000", 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.354e-06 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.601159+0000", 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004586789 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.601261+0000", 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010241799999999999 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.601726+0000", 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00046440199999999999 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.860 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.602186+0000", 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00046048399999999999 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.607457+0000", 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052708670000000003 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.607524+0000", 2024-09-16T03:41:17.861 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7255000000000006e-05 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:892 3.6s0 3:656a4357:::smithi12138290-23:head [delete] snapc 1d=[1d,1b,1a,11,b,8] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.480742+0000", 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 14.263470884, 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014033755, 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.862 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 892 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480742+0000", 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480742+0000", 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480744+0000", 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3439999999999999e-06 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480753+0000", 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0760000000000003e-06 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480769+0000", 2024-09-16T03:41:17.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6079999999999999e-05 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480778+0000", 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4129999999999995e-06 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.482114+0000", 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0013362179999999999 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.482254+0000", 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000140056 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.494776+0000", 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012521568 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.866 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.bs6 84/31 ECSubWrite(tid=137, reqid=osd.10.0:137, at_version=84'66, trim_to=0'0, roll_forward_to=82'64))", 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.481690+0000", 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 14.262522930999999, 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075420139999999997, 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481690+0000", 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.867 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481690+0000", 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481690+0000", 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5900000000000003e-07 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481783+0000", 2024-09-16T03:41:17.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2295999999999999e-05 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481793+0000", 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0363000000000001e-05 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481801+0000", 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1880000000000002e-06 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481862+0000", 2024-09-16T03:41:17.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0949000000000002e-05 2024-09-16T03:41:17.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481873+0000", 2024-09-16T03:41:17.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0847e-05 2024-09-16T03:41:17.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.872 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489183+0000", 2024-09-16T03:41:17.872 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007309505 2024-09-16T03:41:17.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.872 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.872 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.872 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489232+0000", 2024-09-16T03:41:17.872 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9506999999999999e-05 2024-09-16T03:41:17.872 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.bs6 94/31 ECSubWrite(tid=161, reqid=osd.10.0:161, at_version=94'76, trim_to=0'0, roll_forward_to=91'72))", 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.488968+0000", 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.2552454559999999, 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0088018090000000007, 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488968+0000", 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488968+0000", 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488968+0000", 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1799999999999995e-07 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488989+0000", 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1359999999999999e-05 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488998+0000", 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9379999999999998e-06 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.489006+0000", 2024-09-16T03:41:17.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5549999999999997e-06 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.489107+0000", 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000101056 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.489123+0000", 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5637e-05 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.876 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497728+0000", 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086051900000000004 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497769+0000", 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1554999999999998e-05 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:998 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc 26=[26,25,24,23,22,21,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T03:41:17.877 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.482625+0000", 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.2615882169999999, 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.01249548, 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 998 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482625+0000", 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482625+0000", 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482627+0000", 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8309999999999999e-06 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482632+0000", 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.84e-06 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482646+0000", 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3271999999999999e-05 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482653+0000", 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.391e-06 2024-09-16T03:41:17.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482713+0000", 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9732999999999999e-05 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482721+0000", 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2109999999999998e-06 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.881 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484365+0000", 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001643512 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484488+0000", 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000123516 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.485295+0000", 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000807386 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.882 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.494614+0000", 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0093186829999999995 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.495033+0000", 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00041857299999999999 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.495120+0000", 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7532000000000002e-05 2024-09-16T03:41:17.883 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:999 3.6s0 3:64535dd7:::smithi12138290-18:head [rollback 1d] snapc 26=[26,25,24,23,22,21,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.482760+0000", 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.2614530909999999, 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.018206513000000001, 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.884 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.885 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 999 2024-09-16T03:41:17.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.885 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.885 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482760+0000", 2024-09-16T03:41:17.885 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482760+0000", 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482761+0000", 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2399999999999998e-07 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482764+0000", 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.439e-06 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.886 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482771+0000", 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3919999999999997e-06 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482777+0000", 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1e-06 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482822+0000", 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4997999999999998e-05 2024-09-16T03:41:17.887 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482829+0000", 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8820000000000003e-06 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.486550+0000", 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037209370000000001 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.486647+0000", 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6331999999999999e-05 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487297+0000", 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00065033799999999998 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.494616+0000", 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073187460000000001 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.889 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.500895+0000", 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006279438 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.500966+0000", 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1286999999999997e-05 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.890 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:1000 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc 26=[26,25,24,23,22,21,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.482841+0000", 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.2613717520000001, 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.023637195, 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1000 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:17.891 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482841+0000", 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482841+0000", 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482842+0000", 2024-09-16T03:41:17.892 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3200000000000005e-07 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482846+0000", 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6990000000000001e-06 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482852+0000", 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.883e-06 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:17.893 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482858+0000", 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7769999999999999e-06 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482908+0000", 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9469999999999999e-05 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482913+0000", 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.894 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.495570+0000", 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012656514000000001 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.495672+0000", 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102756 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.496146+0000", 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047371199999999999 2024-09-16T03:41:17.895 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.499031+0000", 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028847339999999999 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.506438+0000", 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0074070459999999996 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:17.896 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.506478+0000", 2024-09-16T03:41:17.897 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0608000000000002e-05 2024-09-16T03:41:17.897 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.897 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.897 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.897 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:17.897 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:17.897 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:17.898 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:17.896+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:17.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:41:17.999 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:17.997+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:18.007 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:41:18.008 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:18.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:41:18.100 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:18.098+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:18.119 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:18.119 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:18.119 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:41:18.119 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:41:18.119 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:18.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:41:18.201 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:18.199+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:18.233 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:41:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:41:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:41:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 73/31 ECSubWrite(tid=83, reqid=osd.1.0:83, at_version=73'38, trim_to=0'0, roll_forward_to=72'36))", 2024-09-16T03:41:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.751893198000001, 2024-09-16T03:41:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059263110000000001, 2024-09-16T03:41:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.17e-07 2024-09-16T03:41:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477878+0000", 2024-09-16T03:41:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4824000000000001e-05 2024-09-16T03:41:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477887+0000", 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6509999999999996e-06 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477895+0000", 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.401e-06 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477945+0000", 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0555e-05 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477953+0000", 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.875e-06 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483761+0000", 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058074750000000003 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483789+0000", 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8313000000000001e-05 2024-09-16T03:41:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 73/31 ECSubWrite(tid=85, reqid=osd.1.0:85, at_version=73'40, trim_to=0'0, roll_forward_to=72'36))", 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.748596472999999, 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071150040000000003, 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481160+0000", 2024-09-16T03:41:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481177+0000", 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7368999999999999e-05 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481186+0000", 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1940000000000004e-06 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481194+0000", 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4209999999999998e-06 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481262+0000", 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8312000000000006e-05 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481272+0000", 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3519999999999996e-06 2024-09-16T03:41:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488236+0000", 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069643860000000004 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488274+0000", 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8390999999999999e-05 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=111, reqid=osd.6.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.738141050999999, 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006602218, 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491625+0000", 2024-09-16T03:41:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7249999999999992e-06 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491634+0000", 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3370000000000006e-06 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491642+0000", 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0770000000000002e-06 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491764+0000", 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012253500000000001 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491772+0000", 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1499999999999999e-06 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498186+0000", 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064136640000000003 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498217+0000", 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1080000000000001e-05 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 73/55 ECSubWrite(tid=98, reqid=osd.10.0:98, at_version=73'24, trim_to=0'0, roll_forward_to=56'20))", 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.736275103000001, 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065373489999999996, 2024-09-16T03:41:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.61e-07 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493506+0000", 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4556999999999999e-05 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493520+0000", 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3804e-05 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493528+0000", 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9340000000000006e-06 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493606+0000", 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8144000000000003e-05 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493617+0000", 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1368e-05 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499976+0000", 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006358463 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500018+0000", 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2317999999999997e-05 2024-09-16T03:41:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=112, reqid=osd.6.0:112, at_version=73'91, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.735951106000002, 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062537729999999998, 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5600000000000005e-07 2024-09-16T03:41:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493823+0000", 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7804000000000001e-05 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493835+0000", 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2197e-05 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493845+0000", 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6409999999999997e-06 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493899+0000", 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4418e-05 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493909+0000", 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5670000000000001e-06 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500026+0000", 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006117468 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500058+0000", 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2021999999999999e-05 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=113, reqid=osd.6.0:113, at_version=73'93, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.735668218000001, 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083691979999999996, 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0399999999999995e-07 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494110+0000", 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1750999999999999e-05 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494122+0000", 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1532000000000001e-05 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494129+0000", 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7300000000000005e-06 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495262+0000", 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011327679999999999 2024-09-16T03:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495271+0000", 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7739999999999993e-06 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502425+0000", 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071538160000000003 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502457+0000", 2024-09-16T03:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2122999999999997e-05 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:682 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.702175650000001, 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095978390000000004, 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 682 2024-09-16T03:41:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:41:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527582+0000", 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7910000000000001e-06 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527590+0000", 2024-09-16T03:41:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5320000000000001e-06 2024-09-16T03:41:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527608+0000", 2024-09-16T03:41:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8496999999999999e-05 2024-09-16T03:41:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527622+0000", 2024-09-16T03:41:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3583e-05 2024-09-16T03:41:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527699+0000", 2024-09-16T03:41:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7799000000000005e-05 2024-09-16T03:41:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527707+0000", 2024-09-16T03:41:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:41:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.530874+0000", 2024-09-16T03:41:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031660529999999998 2024-09-16T03:41:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.530988+0000", 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114376 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.531781+0000", 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079327 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536093+0000", 2024-09-16T03:41:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043113819999999999 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.537011+0000", 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091858900000000004 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.537178+0000", 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016688799999999999 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:683 3.es0 3:71ce6426:::smithi12138290-50:head [rollback 18] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.702100937000001, 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015112265, 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 683 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527659+0000", 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4510000000000001e-06 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527667+0000", 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527673+0000", 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0100000000000001e-06 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527773+0000", 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100029 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527780+0000", 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3239999999999999e-06 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.532223+0000", 2024-09-16T03:41:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044434870000000003 2024-09-16T03:41:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.532315+0000", 2024-09-16T03:41:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2255000000000004e-05 2024-09-16T03:41:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533195+0000", 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087904200000000002 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536094+0000", 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028993840000000001 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542627+0000", 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006533021 2024-09-16T03:41:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542767+0000", 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014024 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:684 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.7020719, 2024-09-16T03:41:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015421104, 2024-09-16T03:41:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 684 2024-09-16T03:41:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000005e-07 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527687+0000", 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4609999999999998e-06 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527696+0000", 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7600000000000008e-06 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527701+0000", 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3789999999999997e-06 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527803+0000", 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010234900000000001 2024-09-16T03:41:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527809+0000", 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2870000000000004e-06 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533474+0000", 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056649090000000001 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533571+0000", 2024-09-16T03:41:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7552000000000004e-05 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.534004+0000", 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00043247700000000001 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536095+0000", 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020918460000000001 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542953+0000", 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068572249999999998 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.543105+0000", 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015232000000000001 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=138, reqid=osd.6.0:138, at_version=78'117, trim_to=0'0, roll_forward_to=77'113))", 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:41:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.682496142000002, 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064033809999999997, 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:41:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547261+0000", 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.032e-06 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547287+0000", 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5905e-05 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547301+0000", 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3905e-05 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547310+0000", 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4590000000000004e-06 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547537+0000", 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022751299999999999 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547553+0000", 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5113000000000001e-05 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553619+0000", 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060667450000000001 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553663+0000", 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3708999999999998e-05 2024-09-16T03:41:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 78/55 ECSubWrite(tid=122, reqid=osd.10.0:122, at_version=78'32, trim_to=0'0, roll_forward_to=76'30))", 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.679719022, 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006109504, 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9100000000000004e-07 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550065+0000", 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7671999999999999e-05 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550075+0000", 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5540000000000003e-06 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550083+0000", 2024-09-16T03:41:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8119999999999997e-06 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550154+0000", 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0167999999999996e-05 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550163+0000", 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7850000000000003e-06 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556099+0000", 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059356310000000002 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556146+0000", 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7391000000000001e-05 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=139, reqid=osd.6.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:41:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.675653693000001, 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078423550000000005, 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554103+0000", 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7700000000000004e-07 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554111+0000", 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6119999999999999e-06 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554122+0000", 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0587999999999999e-05 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554129+0000", 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554193+0000", 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3522000000000006e-05 2024-09-16T03:41:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554204+0000", 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.083e-05 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561904+0000", 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077002540000000001 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561944+0000", 2024-09-16T03:41:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0538999999999999e-05 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.as3 78/55 ECSubWrite(tid=161, reqid=osd.12.0:161, at_version=78'43, trim_to=0'0, roll_forward_to=77'41))", 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.674933473999999, 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067129479999999998, 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554823+0000", 2024-09-16T03:41:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554849+0000", 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6461999999999998e-05 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554862+0000", 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2918000000000001e-05 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554872+0000", 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.005e-05 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554971+0000", 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8665999999999994e-05 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554981+0000", 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0161e-05 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561484+0000", 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006503404 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561535+0000", 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0934999999999999e-05 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 78/31 ECSubWrite(tid=133, reqid=osd.1.0:133, at_version=78'56, trim_to=0'0, roll_forward_to=76'52))", 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.674430112, 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006360844, 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555327+0000", 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1299999999999998e-07 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555347+0000", 2024-09-16T03:41:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0389000000000001e-05 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555359+0000", 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2371999999999999e-05 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555368+0000", 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0480000000000004e-06 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555454+0000", 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5662999999999996e-05 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555464+0000", 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0018e-05 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561663+0000", 2024-09-16T03:41:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061989549999999999 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561687+0000", 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3485999999999998e-05 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.as3 78/55 ECSubWrite(tid=163, reqid=osd.12.0:163, at_version=78'45, trim_to=0'0, roll_forward_to=77'41))", 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.673550179999999, 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078281010000000005, 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556207+0000", 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0809999999999999e-06 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556231+0000", 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.406e-05 2024-09-16T03:41:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556244+0000", 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3696999999999999e-05 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556258+0000", 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3342e-05 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556856+0000", 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059772700000000005 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556865+0000", 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.166e-06 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564009+0000", 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007144358 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564034+0000", 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4669999999999999e-05 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=140, reqid=osd.6.0:140, at_version=78'119, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.672140955, 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064168259999999996, 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557616+0000", 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9899999999999999e-07 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557640+0000", 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4034000000000001e-05 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557651+0000", 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1382999999999999e-05 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557660+0000", 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557721+0000", 2024-09-16T03:41:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1484999999999998e-05 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557731+0000", 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0051000000000001e-05 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.563993+0000", 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062610649999999997 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564032+0000", 2024-09-16T03:41:18.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9193999999999998e-05 2024-09-16T03:41:18.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.318 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.318 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:879 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:18.319 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.480957+0000", 2024-09-16T03:41:18.319 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.748798884999999, 2024-09-16T03:41:18.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091711669999999992, 2024-09-16T03:41:18.319 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.319 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 879 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480957+0000", 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480957+0000", 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480959+0000", 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2259999999999999e-06 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480967+0000", 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.401e-06 2024-09-16T03:41:18.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480983+0000", 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6704000000000001e-05 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480993+0000", 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0013000000000001e-05 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481088+0000", 2024-09-16T03:41:18.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4381000000000003e-05 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481098+0000", 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8640000000000008e-06 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.483602+0000", 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025048539999999999 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:18.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.483734+0000", 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000131617 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.484915+0000", 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001180918 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.489925+0000", 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050105779999999999 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490013+0000", 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7682000000000005e-05 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490128+0000", 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114929 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:880 3.es0 3:71ce6426:::smithi12138290-50:head [rollback 8] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:18.325 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.748736059000001, 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010599224000000001, 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 880 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2799999999999995e-07 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481024+0000", 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481036+0000", 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2088e-05 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481044+0000", 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5129999999999999e-06 2024-09-16T03:41:18.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481172+0000", 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012779299999999999 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481177+0000", 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.485371+0000", 2024-09-16T03:41:18.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041941410000000002 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.485473+0000", 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102734 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.486086+0000", 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061263300000000001 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490135+0000", 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040486020000000001 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491526+0000", 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001391155 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491619+0000", 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3103000000000006e-05 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:881 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.748672036, 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010772182999999999, 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:18.332 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 881 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9500000000000003e-07 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481089+0000", 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481096+0000", 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5549999999999997e-06 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481101+0000", 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4800000000000003e-06 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481212+0000", 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111263 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481220+0000", 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4069999999999993e-06 2024-09-16T03:41:18.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.486523+0000", 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053031099999999998 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.486636+0000", 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112339 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.487220+0000", 2024-09-16T03:41:18.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058448099999999998 2024-09-16T03:41:18.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490409+0000", 2024-09-16T03:41:18.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031889729999999999 2024-09-16T03:41:18.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:18.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491777+0000", 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001367458 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491856+0000", 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9231000000000001e-05 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:985 3.es0 3:7acde04a:::smithi12138290-46:head [stat,delete] snapc 25=[25,24,23,22,21,20,1f,1e,1d,1b,b,8] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.482982+0000", 2024-09-16T03:41:18.338 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.746774168, 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010206787, 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 985 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:18.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.482982+0000", 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.482982+0000", 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.482983+0000", 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.567e-06 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.482989+0000", 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1739999999999997e-06 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.483003+0000", 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3555e-05 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.483011+0000", 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1300000000000001e-06 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.483530+0000", 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051879400000000005 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.483537+0000", 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9870000000000002e-06 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.484023+0000", 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000486262 2024-09-16T03:41:18.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.484137+0000", 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011434000000000001 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.484926+0000", 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078816599999999998 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.488760+0000", 2024-09-16T03:41:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003834035 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.493017+0000", 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042577020000000004 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.493188+0000", 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000171075 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:18.345 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:18.345 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:18.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:41:18.347 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:18.344+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:18.374 INFO:teuthology.orchestra.run.smithi076.stdout:fsck success 2024-09-16T03:41:18.382 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-09-16T03:41:18.383 INFO:tasks.thrashosds.thrasher:Reviving osd 12 2024-09-16T03:41:18.383 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-16T03:41:18.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-16T03:41:18.419 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:41:18.425 INFO:tasks.ceph.osd.12:Started 2024-09-16T03:41:18.425 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:41:18.446 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:18.445+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:18.490 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:41:18.490 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:18.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:41:18.513 INFO:tasks.rados.rados.0.smithi121.stdout:386: done (3 left) 2024-09-16T03:41:18.513 INFO:tasks.rados.rados.0.smithi121.stdout:387: done (2 left) 2024-09-16T03:41:18.513 INFO:tasks.rados.rados.0.smithi121.stdout:388: done (1 left) 2024-09-16T03:41:18.513 INFO:tasks.rados.rados.0.smithi121.stdout:389: done (0 left) 2024-09-16T03:41:18.514 INFO:tasks.rados.rados.0.smithi121.stdout:390: rollback oid 14 current snap is 38 2024-09-16T03:41:18.514 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 14 to 33 2024-09-16T03:41:18.514 INFO:tasks.rados.rados.0.smithi121.stdout:391: snap_create 2024-09-16T03:41:18.524 INFO:tasks.rados.rados.0.smithi121.stdout:390: finishing rollback tid 0 to smithi12138290-14 2024-09-16T03:41:18.526 INFO:tasks.rados.rados.0.smithi121.stdout:390: finishing rollback tid 1 to smithi12138290-14 2024-09-16T03:41:18.526 INFO:tasks.rados.rados.0.smithi121.stdout:390: finishing rollback tid 2 to smithi12138290-14 2024-09-16T03:41:18.526 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 14 v 213 (ObjNum 142 snap 28 seq_num 142) dirty exists 2024-09-16T03:41:18.527 INFO:teuthology.orchestra.run.smithi076.stderr:no valid command found; 10 closest matches: 2024-09-16T03:41:18.527 INFO:teuthology.orchestra.run.smithi076.stderr:0 2024-09-16T03:41:18.527 INFO:teuthology.orchestra.run.smithi076.stderr:1 2024-09-16T03:41:18.527 INFO:teuthology.orchestra.run.smithi076.stderr:2 2024-09-16T03:41:18.527 INFO:teuthology.orchestra.run.smithi076.stderr:abort 2024-09-16T03:41:18.527 INFO:teuthology.orchestra.run.smithi076.stderr:assert 2024-09-16T03:41:18.527 INFO:teuthology.orchestra.run.smithi076.stderr:config diff 2024-09-16T03:41:18.527 INFO:teuthology.orchestra.run.smithi076.stderr:config diff get 2024-09-16T03:41:18.527 INFO:teuthology.orchestra.run.smithi076.stderr:config get 2024-09-16T03:41:18.528 INFO:teuthology.orchestra.run.smithi076.stderr:config help [] 2024-09-16T03:41:18.528 INFO:teuthology.orchestra.run.smithi076.stderr:config set ... 2024-09-16T03:41:18.528 INFO:teuthology.orchestra.run.smithi076.stderr:admin_socket: invalid command 2024-09-16T03:41:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:41:18.530 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['dump_ops_in_flight'] 2024-09-16T03:41:18.547 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:41:18.545+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:41:18.550 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:41:18.550 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:41:18.567 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:41:18.568 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:41:18.584 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:41:18.584 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:41:18.600 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:41:18.601 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:18.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:41:18.616 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:41:18.617 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:41:18.633 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:41:18.633 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:41:18.648 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:18.647+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:18.650 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:41:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:41:18.667 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:41:18.667 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:41:18.682 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:41:18.682 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:41:18.699 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:41:18.700 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:41:18.715 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:41:18.716 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:41:18.732 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:41:18.732 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:41:18.732 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-16T03:41:18.749 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:41:18.750 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:41:18.750 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:18.748+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:18.757 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:18.757 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:18.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:41:18.766 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:41:18.766 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:41:18.783 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:41:18.783 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:41:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:41:18.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:41:18.851 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-16T03:41:18.849+0000 7f237ce95640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32985) UID: 0 2024-09-16T03:41:18.909 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:18.909 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:18.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:41:18.952 INFO:tasks.ceph.osd.13.smithi103.stderr:2024-09-16T03:41:18.951+0000 7f159cb33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 43798) UID: 0 2024-09-16T03:41:19.020 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:19.021 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:19.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:41:19.053 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:19.052+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:19.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:41:19.154 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:19.152+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:19.243 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:19.243 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:19.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:41:19.255 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:19.253+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:19.355 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:19.355 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:19.356 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:19.354+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:19.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:41:19.457 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:19.455+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:19.466 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:19.466 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:19.474 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:41:19.496 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:41:19.524 INFO:tasks.rados.rados.0.smithi121.stdout:390: done (1 left) 2024-09-16T03:41:19.525 INFO:tasks.rados.rados.0.smithi121.stdout:391: done (0 left) 2024-09-16T03:41:19.525 INFO:tasks.rados.rados.0.smithi121.stdout:392: copy_from oid 24 from oid 10 current snap is 39 2024-09-16T03:41:19.525 INFO:tasks.rados.rados.0.smithi121.stdout:append oid 12 current snap is 39 2024-09-16T03:41:19.525 INFO:tasks.rados.rados.0.smithi121.stdout:393: seq_num 164 ranges {3424256=360448} 2024-09-16T03:41:19.546 INFO:tasks.rados.rados.0.smithi121.stdout:393: writing smithi12138290-12 from 3424256 to 3784704 tid 1 2024-09-16T03:41:19.546 INFO:tasks.rados.rados.0.smithi121.stdout:394: read oid 27 snap -1 2024-09-16T03:41:19.546 INFO:tasks.rados.rados.0.smithi121.stdout:394: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:41:19.546 INFO:tasks.rados.rados.0.smithi121.stdout:395: delete oid 39 current snap is 39 2024-09-16T03:41:19.546 INFO:tasks.rados.rados.0.smithi121.stdout:392: finishing copy_from racing read to smithi12138290-24 2024-09-16T03:41:19.546 INFO:tasks.rados.rados.0.smithi121.stdout:392: finishing copy_from to smithi12138290-24 2024-09-16T03:41:19.546 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 24 v 16 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-09-16T03:41:19.549 INFO:tasks.rados.rados.0.smithi121.stdout:394: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:41:19.554 INFO:tasks.rados.rados.0.smithi121.stdout:392: done (3 left) 2024-09-16T03:41:19.555 INFO:tasks.rados.rados.0.smithi121.stdout:394: done (2 left) 2024-09-16T03:41:19.555 INFO:tasks.rados.rados.0.smithi121.stdout:395: done (1 left) 2024-09-16T03:41:19.555 INFO:tasks.rados.rados.0.smithi121.stdout:396: snap_remove snap 36 2024-09-16T03:41:19.558 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:19.556+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:19.597 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:19.597 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:19.606 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:41:19.659 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:41:19.656+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:41:19.709 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:19.709 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:19.718 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:41:19.761 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:19.760+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:19.821 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:19.821 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:19.830 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:41:19.860 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:19.859+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:19.933 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:19.933 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:19.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:41:19.985 INFO:tasks.ceph.osd.8.smithi076.stderr:2024-09-16T03:41:19.983+0000 7fdb73429640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33489) UID: 0 2024-09-16T03:41:20.043 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:20.043 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:20.052 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:41:20.092 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:41:20.090+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:41:20.163 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:20.161+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:20.195 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:20.195 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:20.204 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:41:20.263 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:41:20.261+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:41:20.309 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:20.309 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:20.317 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:41:20.357 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:41:20.354+0000 7f58ee745640 -1 Falling back to public interface 2024-09-16T03:41:20.364 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:20.363+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:20.460 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:20.460 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:20.465 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:20.464+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:20.470 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:41:20.526 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T03:41:20.528 INFO:tasks.rados.rados.0.smithi121.stdout:396: done (1 left) 2024-09-16T03:41:20.529 INFO:tasks.rados.rados.0.smithi121.stdout:397: delete oid 22 current snap is 39 2024-09-16T03:41:20.529 INFO:tasks.rados.rados.0.smithi121.stdout:393: finishing write tid 1 to smithi12138290-12 2024-09-16T03:41:20.529 INFO:tasks.rados.rados.0.smithi121.stdout:393: finishing write tid 2 to smithi12138290-12 2024-09-16T03:41:20.529 INFO:tasks.rados.rados.0.smithi121.stdout:393: finishing write tid 3 to smithi12138290-12 2024-09-16T03:41:20.529 INFO:tasks.rados.rados.0.smithi121.stdout:393: oid 12 updating version 0 to 108 2024-09-16T03:41:20.529 INFO:tasks.rados.rados.0.smithi121.stdout:393: oid 12 updating version 108 to 109 2024-09-16T03:41:20.529 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 12 v 109 (ObjNum 164 snap 39 seq_num 164) dirty exists 2024-09-16T03:41:20.529 INFO:tasks.rados.rados.0.smithi121.stdout:393: left oid 12 (ObjNum 164 snap 39 seq_num 164) 2024-09-16T03:41:20.539 INFO:tasks.rados.rados.0.smithi121.stdout:393: done (1 left) 2024-09-16T03:41:20.539 INFO:tasks.rados.rados.0.smithi121.stdout:397: done (0 left) 2024-09-16T03:41:20.539 INFO:tasks.rados.rados.0.smithi121.stdout:398: rollback oid 30 current snap is 39 2024-09-16T03:41:20.539 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 30 to 6 2024-09-16T03:41:20.539 INFO:tasks.rados.rados.0.smithi121.stdout:399: rollback oid 21 current snap is 39 2024-09-16T03:41:20.539 INFO:tasks.rados.rados.0.smithi121.stdout:rollback oid 21 to 32 2024-09-16T03:41:20.540 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 2 2024-09-16T03:41:20.547 INFO:tasks.rados.rados.0.smithi121.stdout:398: finishing rollback tid 0 to smithi12138290-30 2024-09-16T03:41:20.547 INFO:tasks.rados.rados.0.smithi121.stdout:398: finishing rollback tid 1 to smithi12138290-30 2024-09-16T03:41:20.549 INFO:tasks.rados.rados.0.smithi121.stdout:398: finishing rollback tid 2 to smithi12138290-30 2024-09-16T03:41:20.549 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 30 v 219 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:41:20.549 INFO:tasks.rados.rados.0.smithi121.stdout:399: finishing rollback tid 0 to smithi12138290-21 2024-09-16T03:41:20.549 INFO:tasks.rados.rados.0.smithi121.stdout:398: done (1 left) 2024-09-16T03:41:20.549 INFO:tasks.rados.rados.0.smithi121.stdout: waiting on 1 2024-09-16T03:41:20.550 INFO:tasks.rados.rados.0.smithi121.stdout:399: finishing rollback tid 1 to smithi12138290-21 2024-09-16T03:41:20.550 INFO:tasks.rados.rados.0.smithi121.stdout:399: finishing rollback tid 2 to smithi12138290-21 2024-09-16T03:41:20.550 INFO:tasks.rados.rados.0.smithi121.stdout:update_object_version oid 21 v 82 (ObjNum 115 snap 24 seq_num 115) dirty exists 2024-09-16T03:41:20.550 INFO:tasks.rados.rados.0.smithi121.stdout:399: done (0 left) 2024-09-16T03:41:20.554 INFO:tasks.rados.rados.0.smithi121.stderr:0 errors. 2024-09-16T03:41:20.554 INFO:tasks.rados.rados.0.smithi121.stderr: 2024-09-16T03:41:20.558 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:41:20.584 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:20.585 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:20.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:41:20.600 INFO:tasks.ceph.osd.15.smithi121.stderr:2024-09-16T03:41:20.598+0000 7fc6392c1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33197) UID: 0 2024-09-16T03:41:20.666 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:20.664+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:20.740 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:20.740 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:20.749 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:41:20.767 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:20.765+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:20.788 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:41:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:41:20.852 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":101,"fsid":"ecf7a42a-e941-4b24-8693-0ff8b6eb41b1","created":"2024-09-16T03:39:42.686123+0000","modified":"2024-09-16T03:41:20.520427+0000","last_up_change":"2024-09-16T03:41:03.474110+0000","last_in_change":"2024-09-16T03:41:04.475928+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:39:56.952065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:39:57.213209+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":27,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0100002288818359,"score_stable":4.0100002288818359,"optimal_score":0.81000000238418579,"raw_score_acting":3.25,"raw_score_stable":3.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-16T03:40:09.501089+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":40,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"8820e1f9-b619-44a5-abb1-7cfa97fcef00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6811","nonce":2834882614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6813","nonce":2834882614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6817","nonce":2834882614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6815","nonce":2834882614}]},"public_addr":"172.21.15.76:6811/2834882614","cluster_addr":"172.21.15.76:6813/2834882614","heartbeat_back_addr":"172.21.15.76:6817/2834882614","heartbeat_front_addr":"172.21.15.76:6815/2834882614","state":["exists","up"]},{"osd":1,"uuid":"0f58ae45-e2ff-4c4f-9875-7d34ec312cbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6809","nonce":2045143946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6811","nonce":2045143946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6815","nonce":2045143946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6813","nonce":2045143946}]},"public_addr":"172.21.15.103:6809/2045143946","cluster_addr":"172.21.15.103:6811/2045143946","heartbeat_back_addr":"172.21.15.103:6815/2045143946","heartbeat_front_addr":"172.21.15.103:6813/2045143946","state":["exists","up"]},{"osd":2,"uuid":"b369fe92-86b8-45b3-a418-024f0e4becd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6817","nonce":273257286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6819","nonce":273257286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6823","nonce":273257286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6821","nonce":273257286}]},"public_addr":"172.21.15.119:6817/273257286","cluster_addr":"172.21.15.119:6819/273257286","heartbeat_back_addr":"172.21.15.119:6823/273257286","heartbeat_front_addr":"172.21.15.119:6821/273257286","state":["exists","up"]},{"osd":3,"uuid":"b9521bf1-46ef-402c-8472-c1b476280b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6811","nonce":278480145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6815","nonce":278480145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6823","nonce":278480145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6819","nonce":278480145}]},"public_addr":"172.21.15.121:6811/278480145","cluster_addr":"172.21.15.121:6815/278480145","heartbeat_back_addr":"172.21.15.121:6823/278480145","heartbeat_front_addr":"172.21.15.121:6819/278480145","state":["exists","up"]},{"osd":4,"uuid":"15bec6a1-fc13-4822-8cfa-899b83886c10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6803","nonce":3302785479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6805","nonce":3302785479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6809","nonce":3302785479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6807","nonce":3302785479}]},"public_addr":"172.21.15.76:6803/3302785479","cluster_addr":"172.21.15.76:6805/3302785479","heartbeat_back_addr":"172.21.15.76:6809/3302785479","heartbeat_front_addr":"172.21.15.76:6807/3302785479","state":["exists","up"]},{"osd":5,"uuid":"e9b234ec-bfa4-4f72-adab-295e067b2e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6801","nonce":3765230361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6803","nonce":3765230361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6807","nonce":3765230361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6805","nonce":3765230361}]},"public_addr":"172.21.15.103:6801/3765230361","cluster_addr":"172.21.15.103:6803/3765230361","heartbeat_back_addr":"172.21.15.103:6807/3765230361","heartbeat_front_addr":"172.21.15.103:6805/3765230361","state":["exists","up"]},{"osd":6,"uuid":"6d4ae9fb-8d8f-451d-a12f-1265302b03ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6825","nonce":446165121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6827","nonce":446165121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6831","nonce":446165121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6829","nonce":446165121}]},"public_addr":"172.21.15.119:6825/446165121","cluster_addr":"172.21.15.119:6827/446165121","heartbeat_back_addr":"172.21.15.119:6831/446165121","heartbeat_front_addr":"172.21.15.119:6829/446165121","state":["exists","up"]},{"osd":7,"uuid":"2c9456ed-00ec-46aa-814e-f85eb441fc4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6810","nonce":1304791608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6814","nonce":1304791608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6822","nonce":1304791608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6818","nonce":1304791608}]},"public_addr":"172.21.15.121:6810/1304791608","cluster_addr":"172.21.15.121:6814/1304791608","heartbeat_back_addr":"172.21.15.121:6822/1304791608","heartbeat_front_addr":"172.21.15.121:6818/1304791608","state":["exists","up"]},{"osd":8,"uuid":"b618a02a-8628-48f5-bb5f-f7591aafa118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6819","nonce":2630853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6821","nonce":2630853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6825","nonce":2630853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6823","nonce":2630853906}]},"public_addr":"172.21.15.76:6819/2630853906","cluster_addr":"172.21.15.76:6821/2630853906","heartbeat_back_addr":"172.21.15.76:6825/2630853906","heartbeat_front_addr":"172.21.15.76:6823/2630853906","state":["exists","up"]},{"osd":9,"uuid":"dc3e5be7-f012-47e6-ba35-0e3354ca0c18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6817","nonce":2702548675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6819","nonce":2702548675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6823","nonce":2702548675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6821","nonce":2702548675}]},"public_addr":"172.21.15.103:6817/2702548675","cluster_addr":"172.21.15.103:6819/2702548675","heartbeat_back_addr":"172.21.15.103:6823/2702548675","heartbeat_front_addr":"172.21.15.103:6821/2702548675","state":["exists","up"]},{"osd":10,"uuid":"4e70a7a8-7ba8-42ab-a45a-e0564b561619","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6809","nonce":205142394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6811","nonce":205142394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6815","nonce":205142394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6813","nonce":205142394}]},"public_addr":"172.21.15.119:6809/205142394","cluster_addr":"172.21.15.119:6811/205142394","heartbeat_back_addr":"172.21.15.119:6815/205142394","heartbeat_front_addr":"172.21.15.119:6813/205142394","state":["exists","up"]},{"osd":11,"uuid":"d16bc493-335f-41c2-bd68-cbad93c49a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6825","nonce":1551091359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6827","nonce":1551091359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6831","nonce":1551091359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6829","nonce":1551091359}]},"public_addr":"172.21.15.121:6825/1551091359","cluster_addr":"172.21.15.121:6827/1551091359","heartbeat_back_addr":"172.21.15.121:6831/1551091359","heartbeat_front_addr":"172.21.15.121:6829/1551091359","state":["exists","up"]},{"osd":12,"uuid":"a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":55,"down_at":84,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6827","nonce":890992293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6829","nonce":890992293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6833","nonce":890992293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":890992293},{"type":"v1","addr":"172.21.15.76:6831","nonce":890992293}]},"public_addr":"172.21.15.76:6827/890992293","cluster_addr":"172.21.15.76:6829/890992293","heartbeat_back_addr":"172.21.15.76:6833/890992293","heartbeat_front_addr":"172.21.15.76:6831/890992293","state":["exists"]},{"osd":13,"uuid":"61e9204b-8269-4c35-b409-5cfbd5e6abbb","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":53,"up_from":76,"up_thru":31,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6825","nonce":3364003535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6827","nonce":3364003535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6831","nonce":3364003535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6829","nonce":3364003535}]},"public_addr":"172.21.15.103:6825/3364003535","cluster_addr":"172.21.15.103:6827/3364003535","heartbeat_back_addr":"172.21.15.103:6831/3364003535","heartbeat_front_addr":"172.21.15.103:6829/3364003535","state":["exists","up"]},{"osd":14,"uuid":"0bd5338e-eb99-44cc-9f5f-1c8ef903d02c","up":1,"in":1,"weight":0.5100555419921875,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6801","nonce":1147733023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6803","nonce":1147733023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6807","nonce":1147733023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6805","nonce":1147733023}]},"public_addr":"172.21.15.119:6801/1147733023","cluster_addr":"172.21.15.119:6803/1147733023","heartbeat_back_addr":"172.21.15.119:6807/1147733023","heartbeat_front_addr":"172.21.15.119:6805/1147733023","state":["exists","up"]},{"osd":15,"uuid":"f240405e-d268-41cc-bd99-5a2ca71a85a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6801","nonce":3149764080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6803","nonce":3149764080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6807","nonce":3149764080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6805","nonce":3149764080}]},"public_addr":"172.21.15.121:6801/3149764080","cluster_addr":"172.21.15.121:6803/3149764080","heartbeat_back_addr":"172.21.15.121:6807/3149764080","heartbeat_front_addr":"172.21.15.121:6805/3149764080","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.895331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:12.029403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.279946+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:59.887564+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:59.388617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:58.560002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.217756+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:56.344679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:03.346738+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:11.711895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:19.361985+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:54.049910+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T03:39:54.019981+0000","dead_epoch":84},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T03:41:11.065521+0000","dead_epoch":54},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:09.217323+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.187965+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.2","osds":[14,3]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":11,"length":1},{"begin":17,"length":1},{"begin":26,"length":1},{"begin":32,"length":2},{"begin":38,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":38,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:41:20.855 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:20.855 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:20.866 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:20.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:41:20.910 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:41:20.907+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:41:20.917 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T03:41:20.917 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:41:20.938 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T03:41:20.938 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:41:20.955 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T03:41:20.955 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:41:20.969 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:20.968+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:20.973 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T03:41:20.973 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:41:20.989 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T03:41:20.990 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:41:21.006 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T03:41:21.007 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:41:21.022 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T03:41:21.022 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:41:21.035 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:41:21.038 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T03:41:21.038 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:41:21.054 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T03:41:21.055 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:41:21.070 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-16T03:41:21.068+0000 7fb17420e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32986) UID: 0 2024-09-16T03:41:21.070 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8: {} 2024-09-16T03:41:21.071 INFO:teuthology.orchestra.run.smithi076.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:41:21.087 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9: {} 2024-09-16T03:41:21.087 INFO:teuthology.orchestra.run.smithi076.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:41:21.104 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10: {} 2024-09-16T03:41:21.104 INFO:teuthology.orchestra.run.smithi076.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:41:21.121 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11: {} 2024-09-16T03:41:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:41:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-16T03:41:21.139 INFO:teuthology.orchestra.run.smithi076.stdout:osd.13: {} 2024-09-16T03:41:21.139 INFO:teuthology.orchestra.run.smithi076.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:41:21.144 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:21.145 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:21.145 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:41:21.145 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:21.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:41:21.156 INFO:teuthology.orchestra.run.smithi076.stdout:osd.14: {} 2024-09-16T03:41:21.157 INFO:teuthology.orchestra.run.smithi076.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:41:21.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:41:21.154+0000 7f7907fff640 0 -- 172.21.15.76:0/172100446 >> [v2:172.21.15.121:6800/3149764080,v1:172.21.15.121:6801/3149764080] conn(0x7f790821bf10 msgr2=0x7f790821e300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T03:41:21.170 INFO:tasks.ceph.osd.3.smithi121.stderr:2024-09-16T03:41:21.169+0000 7f929cb3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33189) UID: 0 2024-09-16T03:41:21.197 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:41:21.197 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:41:21.209 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-16T03:41:20.930370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":138052147,"num_objects":144,"num_object_clones":90,"num_object_copies":988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":9,"num_read":570,"num_read_kb":166488,"num_write":932,"num_write_kb":159558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":97206291,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1742,"ondisk_log_size":1742,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":679908,"kb_used_data":255708,"kb_used_omap":21,"kb_used_meta":423850,"kb_avail":1320525852,"statfs":{"total":1352914698240,"available":1352218472448,"internally_reserved":0,"allocated":261844992,"data_stored":415697258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22328,"internal_metadata":434022600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32768,"num_objects":2,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":10,"num_write_kb":4928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000920"},"pg_stats":[{"pgid":"3.f","version":"97'105","reported_seq":443,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.511012+0000","last_change":"2024-09-16T03:41:16.511719+0000","last_active":"2024-09-16T03:41:16.511012+0000","last_peered":"2024-09-16T03:41:16.511012+0000","last_clean":"2024-09-16T03:41:16.511012+0000","last_became_active":"2024-09-16T03:40:35.521028+0000","last_became_peered":"2024-09-16T03:40:35.521028+0000","last_unstale":"2024-09-16T03:41:16.511012+0000","last_undegraded":"2024-09-16T03:41:16.511012+0000","last_fullsized":"2024-09-16T03:41:16.511012+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:21.263112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024148167000000002,"stat_sum":{"num_bytes":8585216,"num_objects":10,"num_object_clones":7,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":21,"num_read_kb":2698,"num_write":53,"num_write_kb":8768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6045696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5,11,7,1,2],"acting":[0,8,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"1a","length":"1"},{"start":"20","length":"1"}]},{"pgid":"3.e","version":"95'85","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T03:41:15.601298+0000","last_change":"2024-09-16T03:41:14.499562+0000","last_active":"2024-09-16T03:41:15.601298+0000","last_peered":"2024-09-16T03:41:15.601298+0000","last_clean":"2024-09-16T03:41:15.601298+0000","last_became_active":"2024-09-16T03:40:35.524895+0000","last_became_peered":"2024-09-16T03:40:35.524895+0000","last_unstale":"2024-09-16T03:41:15.601298+0000","last_undegraded":"2024-09-16T03:41:15.601298+0000","last_fullsized":"2024-09-16T03:41:15.601298+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:27.780260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082036690000000002,"stat_sum":{"num_bytes":4046848,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":11,"num_read_kb":6580,"num_write":25,"num_write_kb":4592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":6946816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,5,4,2,9,15],"acting":[3,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"},{"start":"19","length":"2"},{"start":"20","length":"1"}]},{"pgid":"3.d","version":"98'123","reported_seq":131,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-16T03:41:19.523695+0000","last_change":"2024-09-16T03:41:17.497846+0000","last_active":"2024-09-16T03:41:19.523695+0000","last_peered":"2024-09-16T03:41:19.523695+0000","last_clean":"2024-09-16T03:41:19.523695+0000","last_became_active":"2024-09-16T03:41:05.496162+0000","last_became_peered":"2024-09-16T03:41:05.496162+0000","last_unstale":"2024-09-16T03:41:19.523695+0000","last_undegraded":"2024-09-16T03:41:19.523695+0000","last_fullsized":"2024-09-16T03:41:19.523695+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:25.071856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062598130000000004,"stat_sum":{"num_bytes":7536640,"num_objects":9,"num_object_clones":6,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":34,"num_read_kb":16174,"num_write":38,"num_write_kb":10544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":17235968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,9,0,8,2,6],"acting":[14,4,9,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"1a","length":"1"},{"start":"20","length":"2"}]},{"pgid":"3.c","version":"97'221","reported_seq":595,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.524176+0000","last_change":"2024-09-16T03:41:16.525027+0000","last_active":"2024-09-16T03:41:16.524176+0000","last_peered":"2024-09-16T03:41:16.524176+0000","last_clean":"2024-09-16T03:41:16.524176+0000","last_became_active":"2024-09-16T03:41:08.595773+0000","last_became_peered":"2024-09-16T03:41:08.595773+0000","last_unstale":"2024-09-16T03:41:16.524176+0000","last_undegraded":"2024-09-16T03:41:16.524176+0000","last_fullsized":"2024-09-16T03:41:16.524176+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:31.272139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037923579999999998,"stat_sum":{"num_bytes":15826944,"num_objects":20,"num_object_clones":13,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":3,"num_read":62,"num_read_kb":17255,"num_write":89,"num_write_kb":21104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":31129600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,15,1,9,2,6],"acting":[0,10,15,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"1a","length":"1"},{"start":"20","length":"1"}]},{"pgid":"3.b","version":"95'78","reported_seq":390,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T03:41:15.505938+0000","last_change":"2024-09-16T03:41:14.498307+0000","last_active":"2024-09-16T03:41:15.505938+0000","last_peered":"2024-09-16T03:41:15.505938+0000","last_clean":"2024-09-16T03:41:15.505938+0000","last_became_active":"2024-09-16T03:40:11.502708+0000","last_became_peered":"2024-09-16T03:40:11.502708+0000","last_unstale":"2024-09-16T03:41:15.505938+0000","last_undegraded":"2024-09-16T03:41:15.505938+0000","last_fullsized":"2024-09-16T03:41:15.505938+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:32.168074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089416519999999996,"stat_sum":{"num_bytes":5439488,"num_objects":8,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":27,"num_read_kb":4954,"num_write":27,"num_write_kb":5312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11","length":"1"},{"start":"19","length":"2"},{"start":"20","length":"1"}]},{"pgid":"3.a","version":"94'71","reported_seq":87,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T03:41:15.506125+0000","last_change":"2024-09-16T03:41:14.485412+0000","last_active":"2024-09-16T03:41:15.506125+0000","last_peered":"2024-09-16T03:41:15.506125+0000","last_clean":"2024-09-16T03:41:15.506125+0000","last_became_active":"2024-09-16T03:41:05.491080+0000","last_became_peered":"2024-09-16T03:41:05.491080+0000","last_unstale":"2024-09-16T03:41:15.506125+0000","last_undegraded":"2024-09-16T03:41:15.506125+0000","last_fullsized":"2024-09-16T03:41:15.506125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:16.763735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081762100000000004,"stat_sum":{"num_bytes":7929856,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":12345,"num_write":31,"num_write_kb":9952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11649024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,3,9,1,8],"acting":[6,4,7,3,9,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11","length":"1"},{"start":"19","length":"2"},{"start":"20","length":"1"}]},{"pgid":"3.9","version":"95'96","reported_seq":285,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T03:41:14.497824+0000","last_change":"2024-09-16T03:41:14.498465+0000","last_active":"2024-09-16T03:41:14.497824+0000","last_peered":"2024-09-16T03:41:14.497824+0000","last_clean":"2024-09-16T03:41:14.497824+0000","last_became_active":"2024-09-16T03:40:35.522230+0000","last_became_peered":"2024-09-16T03:40:35.522230+0000","last_unstale":"2024-09-16T03:41:14.497824+0000","last_undegraded":"2024-09-16T03:41:14.497824+0000","last_fullsized":"2024-09-16T03:41:14.497824+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:34.528422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010859951,"stat_sum":{"num_bytes":13697024,"num_objects":8,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":15660,"num_write":62,"num_write_kb":14432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4358144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,14,9,4,10,6],"acting":[1,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"},{"start":"19","length":"2"},{"start":"20","length":"1"}]},{"pgid":"3.8","version":"97'44","reported_seq":353,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.496055+0000","last_change":"2024-09-16T03:41:16.497008+0000","last_active":"2024-09-16T03:41:16.496055+0000","last_peered":"2024-09-16T03:41:16.496055+0000","last_clean":"2024-09-16T03:41:16.496055+0000","last_became_active":"2024-09-16T03:40:11.516188+0000","last_became_peered":"2024-09-16T03:40:11.516188+0000","last_unstale":"2024-09-16T03:41:16.496055+0000","last_undegraded":"2024-09-16T03:41:16.496055+0000","last_fullsized":"2024-09-16T03:41:16.496055+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012982674,"stat_sum":{"num_bytes":4505600,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":14,"num_write_kb":4400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,0,5],"acting":[11,10,1,15,14,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"1a","length":"1"},{"start":"20","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":348,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.484633+0000","last_change":"2024-09-16T03:39:58.261761+0000","last_active":"2024-09-16T03:41:16.484633+0000","last_peered":"2024-09-16T03:41:16.484633+0000","last_clean":"2024-09-16T03:41:16.484633+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:41:16.484633+0000","last_undegraded":"2024-09-16T03:41:16.484633+0000","last_fullsized":"2024-09-16T03:41:16.484633+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.713404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":158,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.483346+0000","last_change":"2024-09-16T03:40:00.965641+0000","last_active":"2024-09-16T03:41:16.483346+0000","last_peered":"2024-09-16T03:41:16.483346+0000","last_clean":"2024-09-16T03:41:16.483346+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:41:16.483346+0000","last_undegraded":"2024-09-16T03:41:16.483346+0000","last_fullsized":"2024-09-16T03:41:16.483346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:22.133279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"69'13","reported_seq":330,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.483972+0000","last_change":"2024-09-16T03:41:16.484783+0000","last_active":"2024-09-16T03:41:16.483972+0000","last_peered":"2024-09-16T03:41:16.483972+0000","last_clean":"2024-09-16T03:41:16.483972+0000","last_became_active":"2024-09-16T03:40:11.524110+0000","last_became_peered":"2024-09-16T03:40:11.524110+0000","last_unstale":"2024-09-16T03:41:16.483972+0000","last_undegraded":"2024-09-16T03:41:16.483972+0000","last_fullsized":"2024-09-16T03:41:16.483972+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:22.076808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001181398,"stat_sum":{"num_bytes":2949120,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":3125,"num_write":12,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0,4,8,3,7],"acting":[11,5,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"1a","length":"1"},{"start":"20","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":157,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.486472+0000","last_change":"2024-09-16T03:40:00.334268+0000","last_active":"2024-09-16T03:41:16.486472+0000","last_peered":"2024-09-16T03:41:16.486472+0000","last_clean":"2024-09-16T03:41:16.486472+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:41:16.486472+0000","last_undegraded":"2024-09-16T03:41:16.486472+0000","last_fullsized":"2024-09-16T03:41:16.486472+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:16.834552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"95'195","reported_seq":502,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T03:41:15.505956+0000","last_change":"2024-09-16T03:41:14.497324+0000","last_active":"2024-09-16T03:41:15.505956+0000","last_peered":"2024-09-16T03:41:15.505956+0000","last_clean":"2024-09-16T03:41:15.505956+0000","last_became_active":"2024-09-16T03:40:11.523175+0000","last_became_peered":"2024-09-16T03:40:11.523175+0000","last_unstale":"2024-09-16T03:41:15.505956+0000","last_undegraded":"2024-09-16T03:41:15.505956+0000","last_fullsized":"2024-09-16T03:41:15.505956+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:36.653861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012588494,"stat_sum":{"num_bytes":23265280,"num_objects":18,"num_object_clones":13,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":57,"num_read_kb":26598,"num_write":88,"num_write_kb":23952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11","length":"1"},{"start":"19","length":"2"},{"start":"20","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.490515+0000","last_change":"2024-09-16T03:40:00.965529+0000","last_active":"2024-09-16T03:41:16.490515+0000","last_peered":"2024-09-16T03:41:16.490515+0000","last_clean":"2024-09-16T03:41:16.490515+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:41:16.490515+0000","last_undegraded":"2024-09-16T03:41:16.490515+0000","last_fullsized":"2024-09-16T03:41:16.490515+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"95'101","reported_seq":412,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-16T03:41:14.495368+0000","last_change":"2024-09-16T03:41:14.496211+0000","last_active":"2024-09-16T03:41:14.495368+0000","last_peered":"2024-09-16T03:41:14.495368+0000","last_clean":"2024-09-16T03:41:14.495368+0000","last_became_active":"2024-09-16T03:40:11.523741+0000","last_became_peered":"2024-09-16T03:40:11.523741+0000","last_unstale":"2024-09-16T03:41:14.495368+0000","last_undegraded":"2024-09-16T03:41:14.495368+0000","last_fullsized":"2024-09-16T03:41:14.495368+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:31.418322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087195699999999994,"stat_sum":{"num_bytes":9764864,"num_objects":11,"num_object_clones":7,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":10479,"num_write":59,"num_write_kb":12176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"},{"start":"19","length":"2"},{"start":"20","length":"1"}]},{"pgid":"2.2","version":"27'2","reported_seq":127,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-16T03:41:19.523520+0000","last_change":"2024-09-16T03:40:24.329585+0000","last_active":"2024-09-16T03:41:19.523520+0000","last_peered":"2024-09-16T03:41:19.523520+0000","last_clean":"2024-09-16T03:41:19.523520+0000","last_became_active":"2024-09-16T03:40:24.325471+0000","last_became_peered":"2024-09-16T03:40:24.325471+0000","last_unstale":"2024-09-16T03:41:19.523520+0000","last_undegraded":"2024-09-16T03:41:19.523520+0000","last_fullsized":"2024-09-16T03:41:19.523520+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.3","version":"97'42","reported_seq":377,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.505125+0000","last_change":"2024-09-16T03:41:16.508297+0000","last_active":"2024-09-16T03:41:16.505125+0000","last_peered":"2024-09-16T03:41:16.505125+0000","last_clean":"2024-09-16T03:41:16.505125+0000","last_became_active":"2024-09-16T03:41:05.491311+0000","last_became_peered":"2024-09-16T03:41:05.491311+0000","last_unstale":"2024-09-16T03:41:16.505125+0000","last_undegraded":"2024-09-16T03:41:16.505125+0000","last_fullsized":"2024-09-16T03:41:16.505125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:37.304519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017465656,"stat_sum":{"num_bytes":2670592,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":2456,"num_write":20,"num_write_kb":5680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4718592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,0,1,4,7],"acting":[15,8,2,0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"1a","length":"1"},{"start":"20","length":"1"}]},{"pgid":"3.5","version":"97'93","reported_seq":413,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.514445+0000","last_change":"2024-09-16T03:41:16.515188+0000","last_active":"2024-09-16T03:41:16.514445+0000","last_peered":"2024-09-16T03:41:16.514445+0000","last_clean":"2024-09-16T03:41:16.514445+0000","last_became_active":"2024-09-16T03:40:11.518878+0000","last_became_peered":"2024-09-16T03:40:11.518878+0000","last_unstale":"2024-09-16T03:41:16.514445+0000","last_undegraded":"2024-09-16T03:41:16.514445+0000","last_fullsized":"2024-09-16T03:41:16.514445+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:24.804030+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.027747884,"stat_sum":{"num_bytes":10764288,"num_objects":9,"num_object_clones":6,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":34,"num_read_kb":19212,"num_write":51,"num_write_kb":11792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7,11,10,2,5],"acting":[8,6,7,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"1a","length":"1"},{"start":"20","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":155,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T03:41:15.505820+0000","last_change":"2024-09-16T03:40:01.115542+0000","last_active":"2024-09-16T03:41:15.505820+0000","last_peered":"2024-09-16T03:41:15.505820+0000","last_clean":"2024-09-16T03:41:15.505820+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:41:15.505820+0000","last_undegraded":"2024-09-16T03:41:15.505820+0000","last_fullsized":"2024-09-16T03:41:15.505820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.247142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.045739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"94'42","reported_seq":390,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T03:41:15.505667+0000","last_change":"2024-09-16T03:41:14.494704+0000","last_active":"2024-09-16T03:41:15.505667+0000","last_peered":"2024-09-16T03:41:15.505667+0000","last_clean":"2024-09-16T03:41:15.505667+0000","last_became_active":"2024-09-16T03:41:05.489835+0000","last_became_peered":"2024-09-16T03:41:05.489835+0000","last_unstale":"2024-09-16T03:41:15.505667+0000","last_undegraded":"2024-09-16T03:41:15.505667+0000","last_fullsized":"2024-09-16T03:41:15.505667+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045009739999999996,"stat_sum":{"num_bytes":3620864,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14838,"num_write":14,"num_write_kb":3536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":10649600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,14,3,15,1,5],"acting":[10,7,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11","length":"1"},{"start":"19","length":"2"},{"start":"20","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":167,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.483483+0000","last_change":"2024-09-16T03:41:05.059746+0000","last_active":"2024-09-16T03:41:16.483483+0000","last_peered":"2024-09-16T03:41:16.483483+0000","last_clean":"2024-09-16T03:41:16.483483+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:41:16.483483+0000","last_undegraded":"2024-09-16T03:41:16.483483+0000","last_fullsized":"2024-09-16T03:41:16.483483+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:05.059625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:05.059625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:06.887913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"97'71","reported_seq":409,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.493859+0000","last_change":"2024-09-16T03:41:16.494529+0000","last_active":"2024-09-16T03:41:16.493859+0000","last_peered":"2024-09-16T03:41:16.493859+0000","last_clean":"2024-09-16T03:41:16.493859+0000","last_became_active":"2024-09-16T03:40:35.521050+0000","last_became_peered":"2024-09-16T03:40:35.521050+0000","last_unstale":"2024-09-16T03:41:16.493859+0000","last_undegraded":"2024-09-16T03:41:16.493859+0000","last_fullsized":"2024-09-16T03:41:16.493859+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:20.674102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010694232,"stat_sum":{"num_bytes":3702784,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":6920,"num_write":52,"num_write_kb":4048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4472832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7,5,11,6,0],"acting":[9,4,7,5,11,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"1a","length":"1"},{"start":"20","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":101,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-16T03:41:15.601186+0000","last_change":"2024-09-16T03:41:07.602590+0000","last_active":"2024-09-16T03:41:15.601186+0000","last_peered":"2024-09-16T03:41:15.601186+0000","last_clean":"2024-09-16T03:41:15.601186+0000","last_became_active":"2024-09-16T03:40:35.518463+0000","last_became_peered":"2024-09-16T03:40:35.518463+0000","last_unstale":"2024-09-16T03:41:15.601186+0000","last_undegraded":"2024-09-16T03:41:15.601186+0000","last_fullsized":"2024-09-16T03:41:15.601186+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_clean_scrub_stamp":"2024-09-16T03:41:07.602473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:36.559217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"98'202","reported_seq":513,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-16T03:41:19.523700+0000","last_change":"2024-09-16T03:41:17.502422+0000","last_active":"2024-09-16T03:41:19.523700+0000","last_peered":"2024-09-16T03:41:19.523700+0000","last_clean":"2024-09-16T03:41:19.523700+0000","last_became_active":"2024-09-16T03:40:11.517018+0000","last_became_peered":"2024-09-16T03:40:11.517018+0000","last_unstale":"2024-09-16T03:41:19.523700+0000","last_undegraded":"2024-09-16T03:41:19.523700+0000","last_fullsized":"2024-09-16T03:41:19.523700+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:40:10.480737+0000","last_clean_scrub_stamp":"2024-09-16T03:40:10.480737+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010699813000000001,"stat_sum":{"num_bytes":13156352,"num_objects":18,"num_object_clones":12,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":35,"num_read_kb":7072,"num_write":92,"num_write_kb":12352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,0,1],"acting":[14,7,15,8,11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"1a","length":"1"},{"start":"20","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":165,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-16T03:41:16.486745+0000","last_change":"2024-09-16T03:41:16.470343+0000","last_active":"2024-09-16T03:41:16.486745+0000","last_peered":"2024-09-16T03:41:16.486745+0000","last_clean":"2024-09-16T03:41:16.486745+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:41:16.486745+0000","last_undegraded":"2024-09-16T03:41:16.486745+0000","last_fullsized":"2024-09-16T03:41:16.486745+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:16.470256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:16.470256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:29.234901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":137461760,"num_objects":140,"num_object_clones":90,"num_object_copies":980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":9,"num_read":434,"num_read_kb":166370,"num_write":727,"num_write_kb":156096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":97206272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":255762432,"data_stored":425746432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":564,"internal_metadata":0},"log_size":1582,"ondisk_log_size":1582,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46240,"kb_used_data":15052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325600,"statfs":{"total":96636764160,"available":96589414400,"internally_reserved":0,"allocated":15413248,"data_stored":19228697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,5,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":0,"seq":326417514503,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313236,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40536,"kb_used_data":13500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331304,"statfs":{"total":96636764160,"available":96595255296,"internally_reserved":0,"allocated":13824000,"data_stored":20785408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280531,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58484,"kb_used_data":27296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94313356,"statfs":{"total":96636764160,"available":96576876544,"internally_reserved":0,"allocated":27951104,"data_stored":44046109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280531,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56880,"kb_used_data":25684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94314960,"statfs":{"total":96636764160,"available":96578519040,"internally_reserved":0,"allocated":26300416,"data_stored":44572507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280530,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48092,"kb_used_data":16904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323748,"statfs":{"total":96636764160,"available":96587517952,"internally_reserved":0,"allocated":17309696,"data_stored":26560285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[0,1,3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45124,"kb_used_data":13936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326716,"statfs":{"total":96636764160,"available":96590557184,"internally_reserved":0,"allocated":14270464,"data_stored":22189872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46712,"kb_used_data":15524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325128,"statfs":{"total":96636764160,"available":96588931072,"internally_reserved":0,"allocated":15896576,"data_stored":25634841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,5,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247827,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40288,"kb_used_data":13252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331552,"statfs":{"total":96636764160,"available":96595509248,"internally_reserved":0,"allocated":13570048,"data_stored":19200025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247828,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52268,"kb_used_data":21080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319572,"statfs":{"total":96636764160,"available":96583241728,"internally_reserved":0,"allocated":21585920,"data_stored":36915225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42520,"kb_used_data":11328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329320,"statfs":{"total":96636764160,"available":96593223680,"internally_reserved":0,"allocated":11599872,"data_stored":18340409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247828,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51468,"kb_used_data":20276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320372,"statfs":{"total":96636764160,"available":96584060928,"internally_reserved":0,"allocated":20762624,"data_stored":34064409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,5,6,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41540,"kb_used_data":14508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330300,"statfs":{"total":96636764160,"available":96594227200,"internally_reserved":0,"allocated":14856192,"data_stored":19820379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53016,"kb_used_data":21820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318824,"statfs":{"total":96636764160,"available":96582475776,"internally_reserved":0,"allocated":22343680,"data_stored":38883163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,4,5,6,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247827,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56740,"kb_used_data":25548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":26161152,"data_stored":45455929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14475264,"data_stored":24801280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26562560,"data_stored":43233280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12148736,"data_stored":18366464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12877824,"data_stored":21377024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20164608,"data_stored":36081664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9334784,"data_stored":16474112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24895488,"data_stored":43749376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":19030016,"data_stored":32276480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13139968,"data_stored":18944000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20938752,"data_stored":38576128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":15921152,"data_stored":26083328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24207360,"data_stored":44093440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":10715136,"data_stored":15728640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4902912,"data_stored":6688768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":564,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12455936,"data_stored":20877312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13991936,"data_stored":18395136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:21.211 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-16T03:41:21.211 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-16T03:41:21.258 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:21.258 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:21.258 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:41:21.258 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:41:21.258 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:21.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:41:21.272 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-09-16T03:41:21.270+0000 7f1008f02640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33192) UID: 0 2024-09-16T03:41:21.372 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:41:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:41:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:41:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:251 3.0s0 3:00cc7e13:::smithi12138290-42:head [copy-from ver 4] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:41:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:41:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.812656017000002, 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013120588000000001, 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 251 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556736+0000", 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556738+0000", 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8989999999999999e-06 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556745+0000", 2024-09-16T03:41:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:41:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556759+0000", 2024-09-16T03:41:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.451e-05 2024-09-16T03:41:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556766+0000", 2024-09-16T03:41:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1260000000000003e-06 2024-09-16T03:41:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556817+0000", 2024-09-16T03:41:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1693999999999997e-05 2024-09-16T03:41:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:41:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.556824+0000", 2024-09-16T03:41:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8549999999999996e-06 2024-09-16T03:41:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.557940+0000", 2024-09-16T03:41:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001115888 2024-09-16T03:41:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.558009+0000", 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8707999999999993e-05 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.564426+0000", 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064170659999999999 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.566682+0000", 2024-09-16T03:41:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022565039999999999 2024-09-16T03:41:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569779+0000", 2024-09-16T03:41:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030962770000000001 2024-09-16T03:41:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569856+0000", 2024-09-16T03:41:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7843999999999995e-05 2024-09-16T03:41:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.3s4 73/31 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=73'31, trim_to=0'0, roll_forward_to=66'29))", 2024-09-16T03:41:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.484581+0000", 2024-09-16T03:41:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.884810871999999, 2024-09-16T03:41:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012587574000000001, 2024-09-16T03:41:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484581+0000", 2024-09-16T03:41:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484581+0000", 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484581+0000", 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1999999999999999e-07 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484597+0000", 2024-09-16T03:41:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6387000000000001e-05 2024-09-16T03:41:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484607+0000", 2024-09-16T03:41:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1989999999999995e-06 2024-09-16T03:41:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484613+0000", 2024-09-16T03:41:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.879e-06 2024-09-16T03:41:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484707+0000", 2024-09-16T03:41:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4692999999999996e-05 2024-09-16T03:41:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.484723+0000", 2024-09-16T03:41:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5982999999999999e-05 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.497117+0000", 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012393665 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.497169+0000", 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1647999999999997e-05 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.bs4 73/31 ECSubWrite(tid=93, reqid=osd.10.0:93, at_version=73'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.485256+0000", 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.884136096999999, 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012341438999999999, 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485256+0000", 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485256+0000", 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485257+0000", 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.569999999999999e-07 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485274+0000", 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6991999999999999e-05 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485283+0000", 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7110000000000007e-06 2024-09-16T03:41:21.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485289+0000", 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8170000000000004e-06 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485346+0000", 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7258e-05 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.485357+0000", 2024-09-16T03:41:21.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0173999999999999e-05 2024-09-16T03:41:21.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.497566+0000", 2024-09-16T03:41:21.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012209108 2024-09-16T03:41:21.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.497597+0000", 2024-09-16T03:41:21.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1421999999999998e-05 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.bs4 73/31 ECSubWrite(tid=94, reqid=osd.10.0:94, at_version=73'42, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.486560+0000", 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.882831539000001, 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011849755, 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.396 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486560+0000", 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486560+0000", 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486561+0000", 2024-09-16T03:41:21.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2399999999999998e-07 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486579+0000", 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7853e-05 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486589+0000", 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0108999999999999e-05 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.486594+0000", 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1390000000000003e-06 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.489863+0000", 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032689709999999999 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.489873+0000", 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0205e-05 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.498379+0000", 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008506006 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.498410+0000", 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0848e-05 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.400 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.bs4 73/31 ECSubWrite(tid=95, reqid=client.4615.0:649, at_version=73'44, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.488312+0000", 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.881080022999999, 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012984793, 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488312+0000", 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488312+0000", 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488312+0000", 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3300000000000003e-07 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488587+0000", 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00027490799999999998 2024-09-16T03:41:21.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488601+0000", 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3732e-05 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.488610+0000", 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9590000000000001e-06 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493693+0000", 2024-09-16T03:41:21.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050829040000000001 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493702+0000", 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0850000000000003e-06 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.501267+0000", 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075654490000000001 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.501297+0000", 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9323e-05 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:670 3.9s0 3:922516b8:::smithi12138290-20:head [setxattr _header (57) in=7b] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.565823+0000", 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.803568958, 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012597641999999999, 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.405 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 670 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565823+0000", 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565823+0000", 2024-09-16T03:41:21.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565824+0000", 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7100000000000001e-07 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565826+0000", 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565833+0000", 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0329999999999997e-06 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565837+0000", 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9280000000000001e-06 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568016+0000", 2024-09-16T03:41:21.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021785379999999998 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568074+0000", 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8239000000000003e-05 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568410+0000", 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00033627100000000002 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.571764+0000", 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033534540000000001 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578288+0000", 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065244969999999998 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578421+0000", 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013207 2024-09-16T03:41:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:671 3.9s0 3:922516b8:::smithi12138290-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.565843+0000", 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.803548938999999, 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014185764999999999, 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.411 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 671 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565843+0000", 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565843+0000", 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565843+0000", 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1100000000000002e-07 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565845+0000", 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6649999999999999e-06 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565849+0000", 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2039999999999999e-06 2024-09-16T03:41:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.565852+0000", 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0139999999999999e-06 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568727+0000", 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028753720000000002 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568777+0000", 2024-09-16T03:41:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9438000000000003e-05 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578785+0000", 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010007771 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578870+0000", 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5657999999999999e-05 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.580029+0000", 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001158332 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:739 3.9s0 3:90188821:::smithi12138290-22:head [copy-from ver 76] snapc 1b=[1b,1a,19,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:56.475250+0000", 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.894141896000001, 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022129930999999999, 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.416 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.417 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.417 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.417 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.417 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 739 2024-09-16T03:41:21.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.417 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475250+0000", 2024-09-16T03:41:21.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475250+0000", 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475251+0000", 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5710000000000001e-06 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475259+0000", 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5290000000000003e-06 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475272+0000", 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2611e-05 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475279+0000", 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1790000000000002e-06 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475329+0000", 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9895000000000003e-05 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475334+0000", 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4779999999999999e-06 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.477288+0000", 2024-09-16T03:41:21.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019541319999999999 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.477384+0000", 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5963000000000003e-05 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.492225+0000", 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014841127000000001 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.494445+0000", 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022194129999999999 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.497324+0000", 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028792800000000001 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.497380+0000", 2024-09-16T03:41:21.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5753e-05 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:740 3.9s0 3:90188821:::smithi12138290-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:56.475294+0000", 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.894097890000001, 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018563797, 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.423 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 740 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475294+0000", 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475294+0000", 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475294+0000", 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2599999999999998e-07 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475296+0000", 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.66e-06 2024-09-16T03:41:21.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475302+0000", 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9410000000000004e-06 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475309+0000", 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4259999999999997e-06 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475377+0000", 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7725000000000006e-05 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.475382+0000", 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2689999999999999e-06 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.477513+0000", 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00213043 2024-09-16T03:41:21.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.477575+0000", 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1952999999999995e-05 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.493676+0000", 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016101790000000001 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.493762+0000", 2024-09-16T03:41:21.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5504e-05 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.493858+0000", 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5773000000000006e-05 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:764 3.9s0 3:9b8813dd:::smithi12138290-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:56.544476+0000", 2024-09-16T03:41:21.429 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.824915656000002, 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012277615, 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 764 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544476+0000", 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544476+0000", 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544477+0000", 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1799999999999995e-07 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544480+0000", 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.016e-06 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544487+0000", 2024-09-16T03:41:21.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2119999999999997e-06 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.544492+0000", 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0509999999999996e-06 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.550597+0000", 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006104936 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:21.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.550679+0000", 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1734e-05 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.555023+0000", 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043442339999999998 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.555108+0000", 2024-09-16T03:41:21.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4781e-05 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:56.556754+0000", 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016461329999999999 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 84/31 ECSubWrite(tid=162, reqid=osd.14.0:161, at_version=0'0, trim_to=0'0, roll_forward_to=84'152))", 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:03.502046+0000", 2024-09-16T03:41:21.435 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.867346197, 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.027165196999999999, 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502046+0000", 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502046+0000", 2024-09-16T03:41:21.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502046+0000", 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1500000000000004e-07 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502057+0000", 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0873000000000001e-05 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502074+0000", 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6577000000000001e-05 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502082+0000", 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2150000000000001e-06 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502137+0000", 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5315000000000001e-05 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502147+0000", 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2720000000000003e-06 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.529158+0000", 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.027011029999999998 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.529211+0000", 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3199999999999999e-05 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.439 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.6s6 84/31 ECSubWrite(tid=163, reqid=osd.14.0:163, at_version=84'154, trim_to=0'0, roll_forward_to=84'152))", 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:03.502738+0000", 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.866653979999999, 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.026897858, 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502738+0000", 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502738+0000", 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502739+0000", 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8100000000000002e-07 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502760+0000", 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1897000000000001e-05 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502775+0000", 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4651000000000001e-05 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502783+0000", 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0770000000000008e-06 2024-09-16T03:41:21.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502828+0000", 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4549999999999999e-05 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.502833+0000", 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1399999999999999e-06 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.529606+0000", 2024-09-16T03:41:21.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.026773419999999999 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.529636+0000", 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9442e-05 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 84/55 ECSubWriteReply(tid=176, last_complete=84'78, committed=1, applied=1))", 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:03.505235+0000", 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.864156993000002, 2024-09-16T03:41:21.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023252467999999998, 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.505235+0000", 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.505235+0000", 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.505235+0000", 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5799999999999999e-07 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.505239+0000", 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.527759+0000", 2024-09-16T03:41:21.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022520289999999998 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.527775+0000", 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5532999999999998e-05 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.527997+0000", 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022274 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.528487+0000", 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00049001700000000004 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s5 84/31 ECSubWrite(tid=179, reqid=osd.6.0:178, at_version=0'0, trim_to=0'0, roll_forward_to=84'158))", 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:03.505792+0000", 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.863600314999999, 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.02431869, 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.505792+0000", 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.505792+0000", 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.505792+0000", 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2900000000000003e-07 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.505802+0000", 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7070000000000004e-06 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.527765+0000", 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021963376 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.527778+0000", 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2662e-05 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.528526+0000", 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00074810899999999995 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.528534+0000", 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.52e-06 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.530088+0000", 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015543779999999999 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.530110+0000", 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2209e-05 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.452 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:948 3.9s0 3:9b8813dd:::smithi12138290-12:head [copy-from ver 51] snapc 21=[21,20,1f,1e,1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:07.588280+0000", 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.781111643999999, 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017822029, 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 948 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588280+0000", 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588280+0000", 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588282+0000", 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4899999999999999e-06 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588293+0000", 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1148999999999999e-05 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588308+0000", 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4831000000000001e-05 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588315+0000", 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0620000000000003e-06 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588381+0000", 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6705999999999995e-05 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588391+0000", 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3440000000000007e-06 2024-09-16T03:41:21.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.589927+0000", 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015359810000000001 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.589996+0000", 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9244000000000003e-05 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.599199+0000", 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0092032220000000005 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.602458+0000", 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032589699999999999 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.606047+0000", 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035885380000000001 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.606102+0000", 2024-09-16T03:41:21.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5492000000000003e-05 2024-09-16T03:41:21.459 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.459 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.459 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.459 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:949 3.9s0 3:9b8813dd:::smithi12138290-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e88)", 2024-09-16T03:41:21.459 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:07.588328+0000", 2024-09-16T03:41:21.459 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.781063325, 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013738547, 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 949 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588328+0000", 2024-09-16T03:41:21.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588328+0000", 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588329+0000", 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3799999999999998e-07 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588333+0000", 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6689999999999999e-06 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588339+0000", 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7240000000000003e-06 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588343+0000", 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3110000000000001e-06 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588438+0000", 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5614999999999998e-05 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.588442+0000", 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0249999999999996e-06 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.590126+0000", 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016839559999999999 2024-09-16T03:41:21.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.590160+0000", 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3578000000000003e-05 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.601834+0000", 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011674591 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.601937+0000", 2024-09-16T03:41:21.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000102101 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:07.602067+0000", 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013053899999999999 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.cs3 94/88 ECSubWrite(tid=386, reqid=osd.0.0:386, at_version=94'193, trim_to=0'0, roll_forward_to=93'190))", 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.491479+0000", 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.8779131900000001, 2024-09-16T03:41:21.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012852306000000001, 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491479+0000", 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491479+0000", 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491479+0000", 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0900000000000002e-07 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491497+0000", 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8258999999999999e-05 2024-09-16T03:41:21.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491507+0000", 2024-09-16T03:41:21.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9329999999999996e-06 2024-09-16T03:41:21.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491515+0000", 2024-09-16T03:41:21.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4649999999999997e-06 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491585+0000", 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0170000000000001e-05 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491595+0000", 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8800000000000003e-06 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.504308+0000", 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012713221 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.504331+0000", 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2869000000000001e-05 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:995 3.9s0 3:90188821:::smithi12138290-22:head [append 0~0] snapc 26=[26,25,24,23,22,21,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:16.482425+0000", 2024-09-16T03:41:21.470 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.8869666790000004, 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021229649, 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 995 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482425+0000", 2024-09-16T03:41:21.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482425+0000", 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482427+0000", 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.733e-06 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482431+0000", 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3440000000000003e-06 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482449+0000", 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7816e-05 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482458+0000", 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3309999999999993e-06 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482510+0000", 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482627+0000", 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116539 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.483342+0000", 2024-09-16T03:41:21.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071499300000000003 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486888+0000", 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035464569999999998 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.503576+0000", 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016687556999999999 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.503655+0000", 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8801000000000007e-05 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:996 3.9s0 3:90188821:::smithi12138290-22:head [rollback 22] snapc 26=[26,25,24,23,22,21,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:16.482483+0000", 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.8869083920000005, 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021326080000000001, 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.476 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 996 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482483+0000", 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482483+0000", 2024-09-16T03:41:21.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482484+0000", 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4600000000000004e-07 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482488+0000", 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.473e-06 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482497+0000", 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3859999999999996e-06 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482503+0000", 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2160000000000003e-06 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.483985+0000", 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001481584 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.484087+0000", 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010189800000000001 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.484584+0000", 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00049722199999999996 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.490033+0000", 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054490040000000003 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.503746+0000", 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013713088999999999 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.503810+0000", 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3561999999999999e-05 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.481 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.482 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:997 3.9s0 3:90188821:::smithi12138290-22:head [append 0~0] snapc 26=[26,25,24,23,22,21,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T03:41:21.482 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:16.482513+0000", 2024-09-16T03:41:21.482 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.8868784290000002, 2024-09-16T03:41:21.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021415759999999999, 2024-09-16T03:41:21.482 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 997 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482513+0000", 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482513+0000", 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482514+0000", 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7700000000000004e-07 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482516+0000", 2024-09-16T03:41:21.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4940000000000002e-06 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482530+0000", 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3740000000000001e-05 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.482534+0000", 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9999999999999998e-06 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486168+0000", 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036338260000000002 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486264+0000", 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5966999999999998e-05 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486701+0000", 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000436852 2024-09-16T03:41:21.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.492754+0000", 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060532160000000002 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.503882+0000", 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011127814999999999 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.503929+0000", 2024-09-16T03:41:21.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7273e-05 2024-09-16T03:41:21.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.488 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.488 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.488 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:21.489 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:21.369+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:21.490 INFO:teuthology.orchestra.run.smithi076.stdout:osd.15: {} 2024-09-16T03:41:21.491 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:41:21.491 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-09-16T03:41:21.488+0000 7f8f961a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33186) UID: 0 2024-09-16T03:41:21.491 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T03:41:21.491 INFO:teuthology.orchestra.run.smithi076.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:41:21.492 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T03:41:21.590 INFO:tasks.ceph.osd.1.smithi103.stderr:2024-09-16T03:41:21.589+0000 7fd2496cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33184) UID: 0 2024-09-16T03:41:21.635 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:21.636 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:21.636 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:41:21.636 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:21.644 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:41:21.691 INFO:tasks.ceph.osd.10.smithi119.stderr:2024-09-16T03:41:21.689+0000 7f86b0404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32991) UID: 0 2024-09-16T03:41:21.788 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:21.788 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:21.788 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:41:21.789 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:41:21.789 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:21.792 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:41:21.790+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:41:21.797 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:41:21.893 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:21.892+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:21.903 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:21.903 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:41:21.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:41:21.903 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:41:21.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.903 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:179 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'129, mlcod=25'125)", 2024-09-16T03:41:21.903 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:41:21.903 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.616441898000005, 2024-09-16T03:41:21.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066372109999999996, 2024-09-16T03:41:21.903 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284109+0000", 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284110+0000", 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.89e-07 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284175+0000", 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5667000000000007e-05 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284186+0000", 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.009e-05 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284190+0000", 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7600000000000002e-06 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284222+0000", 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1888999999999999e-05 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284234+0000", 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.225e-05 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290722+0000", 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064871909999999998 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.290746+0000", 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4875000000000002e-05 2024-09-16T03:41:21.907 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:180 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'130, mlcod=25'125)", 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.615827589000006, 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072760109999999998, 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284724+0000", 2024-09-16T03:41:21.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.68e-07 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284729+0000", 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4009999999999999e-06 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284738+0000", 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6940000000000007e-06 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.284743+0000", 2024-09-16T03:41:21.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5549999999999996e-06 2024-09-16T03:41:21.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286547+0000", 2024-09-16T03:41:21.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0018044770000000001 2024-09-16T03:41:21.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286567+0000", 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9272000000000001e-05 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.291958+0000", 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0053911540000000004 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292000+0000", 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.189e-05 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:181 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'131, mlcod=25'126)", 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.615287038999995, 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067648710000000004, 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.914 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285264+0000", 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285265+0000", 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6100000000000001e-07 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285269+0000", 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5020000000000003e-06 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285279+0000", 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6879999999999994e-06 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285286+0000", 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4590000000000001e-06 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286842+0000", 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015560000000000001 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.286854+0000", 2024-09-16T03:41:21.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1959e-05 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292001+0000", 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051468859999999998 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292029+0000", 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7815999999999999e-05 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.920 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4257.0:182 1.0 e25/24 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'132, mlcod=25'126)", 2024-09-16T03:41:21.920 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:41:21.920 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.614842319000005, 2024-09-16T03:41:21.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0063484680000000003, 2024-09-16T03:41:21.920 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285709+0000", 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2200000000000004e-07 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285737+0000", 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7886999999999999e-05 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285747+0000", 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9459999999999993e-06 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.285752+0000", 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287024+0000", 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012715409999999999 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.287042+0000", 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8919999999999998e-05 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292031+0000", 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049888179999999999 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:39:59.292057+0000", 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6057000000000001e-05 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 42/31 ECSubWrite(tid=74, reqid=osd.0.0:73, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.337676721999998, 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064234579999999999, 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.925 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.926 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.926 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562874+0000", 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562875+0000", 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562885+0000", 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0069e-05 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562897+0000", 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1619e-05 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562903+0000", 2024-09-16T03:41:21.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.652e-06 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562965+0000", 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1576999999999997e-05 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.562975+0000", 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0314e-05 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569190+0000", 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062148569999999998 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.569298+0000", 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010764600000000001 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.930 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es2 73/55 ECSubWrite(tid=25, reqid=osd.3.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=73'39))", 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.490062+0000", 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.410488937, 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080021980000000003, 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490062+0000", 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490062+0000", 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490063+0000", 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9999999999999996e-07 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490078+0000", 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4795e-05 2024-09-16T03:41:21.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490093+0000", 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4701999999999999e-05 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490100+0000", 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8529999999999992e-06 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490179+0000", 2024-09-16T03:41:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8528000000000004e-05 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490188+0000", 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5219999999999996e-06 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.498003+0000", 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078146499999999994 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.498064+0000", 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1347999999999996e-05 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.8s6 73/31 ECSubWrite(tid=209, reqid=osd.11.0:208, at_version=0'0, trim_to=0'0, roll_forward_to=73'21))", 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.490624+0000", 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.409927410000002, 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074529189999999997, 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.935 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490624+0000", 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490624+0000", 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490625+0000", 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2200000000000003e-07 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490635+0000", 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0237e-05 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490651+0000", 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.664e-05 2024-09-16T03:41:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490661+0000", 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1989999999999995e-06 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490729+0000", 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7939000000000001e-05 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490739+0000", 2024-09-16T03:41:21.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0569e-05 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.498020+0000", 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072808480000000004 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.498077+0000", 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6665e-05 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.5s6 73/31 ECSubWrite(tid=58, reqid=osd.8.0:57, at_version=0'0, trim_to=0'0, roll_forward_to=73'44))", 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.490641+0000", 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.409910027999999, 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074458299999999996, 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490641+0000", 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490641+0000", 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490642+0000", 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3e-07 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490652+0000", 2024-09-16T03:41:21.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0458e-05 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490661+0000", 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7139999999999999e-06 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490666+0000", 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3329999999999996e-06 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490724+0000", 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7836e-05 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.490735+0000", 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0869e-05 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.498011+0000", 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007275643 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.498087+0000", 2024-09-16T03:41:21.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6347000000000006e-05 2024-09-16T03:41:21.944 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.944 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.944 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.944 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.4s6 73/55 ECSubWrite(tid=98, reqid=osd.10.0:98, at_version=73'24, trim_to=0'0, roll_forward_to=56'20))", 2024-09-16T03:41:21.944 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.493643+0000", 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.406908408, 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072825390000000002, 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493643+0000", 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493643+0000", 2024-09-16T03:41:21.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493644+0000", 2024-09-16T03:41:21.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:41:21.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493661+0000", 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7484000000000001e-05 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493673+0000", 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1612e-05 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493681+0000", 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1079999999999993e-06 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493731+0000", 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0005e-05 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.493738+0000", 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.678e-06 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.500897+0000", 2024-09-16T03:41:21.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071581470000000001 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.500925+0000", 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8745999999999999e-05 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 78/55 ECSubWrite(tid=237, reqid=osd.0.0:233, at_version=0'0, trim_to=0'0, roll_forward_to=78'47))", 2024-09-16T03:41:21.949 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:57.568753+0000", 2024-09-16T03:41:21.950 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.33179844, 2024-09-16T03:41:21.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074064409999999997, 2024-09-16T03:41:21.950 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.950 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.950 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.568753+0000", 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.568753+0000", 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.568753+0000", 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6599999999999995e-07 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.568764+0000", 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1015999999999999e-05 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.568777+0000", 2024-09-16T03:41:21.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2903e-05 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.568785+0000", 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6340000000000001e-06 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.568849+0000", 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3664999999999994e-05 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.568861+0000", 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2476e-05 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.576121+0000", 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072600700000000004 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.576159+0000", 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910999999999997e-05 2024-09-16T03:41:21.954 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es2 83/55 ECSubWrite(tid=49, reqid=client.4615.0:881, at_version=0'0, trim_to=0'0, roll_forward_to=83'69))", 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:02.492103+0000", 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.408447894999998, 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077981550000000002, 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.955 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:02.492103+0000", 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:02.492103+0000", 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:02.492104+0000", 2024-09-16T03:41:21.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4900000000000004e-07 2024-09-16T03:41:21.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:02.492116+0000", 2024-09-16T03:41:21.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1844999999999999e-05 2024-09-16T03:41:21.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:02.492131+0000", 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5126999999999999e-05 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:02.492141+0000", 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5170000000000005e-06 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:02.492204+0000", 2024-09-16T03:41:21.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2942999999999997e-05 2024-09-16T03:41:21.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:02.492210+0000", 2024-09-16T03:41:21.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3740000000000003e-06 2024-09-16T03:41:21.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:02.499869+0000", 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076588059999999998 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:02.499901+0000", 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2594e-05 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.960 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.5s6 84/31 ECSubWrite(tid=88, reqid=osd.8.0:88, at_version=84'69, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:03.484433+0000", 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.416118580999999, 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084994979999999994, 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.484433+0000", 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.484433+0000", 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.484433+0000", 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.29e-07 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.484446+0000", 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3159000000000001e-05 2024-09-16T03:41:21.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.484453+0000", 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4460000000000004e-06 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.484459+0000", 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8119999999999996e-06 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.484500+0000", 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.121e-05 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.484506+0000", 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.801e-06 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.492894+0000", 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083881089999999995 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.492932+0000", 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7832e-05 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.5s6 84/31 ECSubWrite(tid=89, reqid=osd.8.0:89, at_version=84'71, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:03.485338+0000", 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.415212668999999, 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080455059999999991, 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485338+0000", 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485338+0000", 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485339+0000", 2024-09-16T03:41:21.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7100000000000001e-07 2024-09-16T03:41:21.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485358+0000", 2024-09-16T03:41:21.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8638e-05 2024-09-16T03:41:21.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485368+0000", 2024-09-16T03:41:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0051000000000001e-05 2024-09-16T03:41:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485376+0000", 2024-09-16T03:41:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7190000000000001e-06 2024-09-16T03:41:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485444+0000", 2024-09-16T03:41:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8847e-05 2024-09-16T03:41:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485454+0000", 2024-09-16T03:41:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2609999999999999e-06 2024-09-16T03:41:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.493349+0000", 2024-09-16T03:41:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078953419999999996 2024-09-16T03:41:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.493384+0000", 2024-09-16T03:41:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4977000000000001e-05 2024-09-16T03:41:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es2 84/55 ECSubWrite(tid=51, reqid=osd.3.0:51, at_version=84'73, trim_to=0'0, roll_forward_to=83'69))", 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:03.485839+0000", 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.414711748999999, 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075410390000000002, 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485839+0000", 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485839+0000", 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485840+0000", 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3300000000000001e-07 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485861+0000", 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1016999999999999e-05 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485875+0000", 2024-09-16T03:41:21.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3451999999999999e-05 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485880+0000", 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.767e-06 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485921+0000", 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0550999999999999e-05 2024-09-16T03:41:21.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.485927+0000", 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.536e-06 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.493349+0000", 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074220589999999999 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.493380+0000", 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1823999999999999e-05 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.8s6 84/31 ECSubWrite(tid=220, reqid=osd.11.0:220, at_version=84'33, trim_to=0'0, roll_forward_to=83'31))", 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:03.489105+0000", 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.411446599000001, 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067452370000000003, 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.977 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.489105+0000", 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.489105+0000", 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.489105+0000", 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8299999999999998e-07 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.489122+0000", 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7328000000000001e-05 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.489131+0000", 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0520000000000007e-06 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.489140+0000", 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5990000000000002e-06 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.489191+0000", 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1425999999999999e-05 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.489197+0000", 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8139999999999997e-06 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.495818+0000", 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066212509999999999 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:03.495850+0000", 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1483999999999999e-05 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.7s3 94/55 ECSubWrite(tid=85, reqid=osd.9.0:84, at_version=0'0, trim_to=0'0, roll_forward_to=94'67))", 2024-09-16T03:41:21.981 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.499414+0000", 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.4011368169999994, 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069666679999999996, 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.499414+0000", 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.499414+0000", 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.499416+0000", 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0949999999999998e-06 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.499435+0000", 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8892000000000001e-05 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.499452+0000", 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6948000000000001e-05 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.499463+0000", 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1191e-05 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.499533+0000", 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9224e-05 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.499568+0000", 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4913999999999998e-05 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.506351+0000", 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067832480000000004 2024-09-16T03:41:21.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.506381+0000", 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0156000000000001e-05 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.5s6 94/31 ECSubWrite(tid=103, reqid=client.4615.0:982, at_version=94'81, trim_to=0'0, roll_forward_to=94'79))", 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.519640+0000", 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.3809113209999992, 2024-09-16T03:41:21.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064134830000000002, 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.519640+0000", 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.519640+0000", 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.519641+0000", 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.122e-06 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.524050+0000", 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044089309999999996 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.524070+0000", 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9809999999999998e-05 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.524080+0000", 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0621e-05 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.524138+0000", 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7247000000000003e-05 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.524146+0000", 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0350000000000001e-06 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.526025+0000", 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001879864 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.526053+0000", 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7852999999999999e-05 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.990 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.7s3 97/55 ECSubWrite(tid=89, reqid=osd.9.0:89, at_version=97'71, trim_to=0'0, roll_forward_to=95'69))", 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:16.485123+0000", 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.4154280420000003, 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083830160000000001, 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.485123+0000", 2024-09-16T03:41:21.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.485123+0000", 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.485123+0000", 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.03e-07 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.485144+0000", 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0088000000000001e-05 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.485157+0000", 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3560999999999999e-05 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.485167+0000", 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5929999999999995e-06 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.485250+0000", 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3218000000000002e-05 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.485261+0000", 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0845e-05 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.493472+0000", 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082116870000000005 2024-09-16T03:41:21.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.493506+0000", 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3720999999999998e-05 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es2 97/55 ECSubWrite(tid=68, reqid=osd.3.0:68, at_version=97'89, trim_to=0'0, roll_forward_to=95'85))", 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:16.486484+0000", 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.4140673750000001, 2024-09-16T03:41:21.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007488481, 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486484+0000", 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486484+0000", 2024-09-16T03:41:21.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:21.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:21.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486485+0000", 2024-09-16T03:41:21.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9899999999999999e-07 2024-09-16T03:41:21.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:21.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486506+0000", 2024-09-16T03:41:21.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0979e-05 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486518+0000", 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2734999999999999e-05 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486528+0000", 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2380000000000003e-06 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486589+0000", 2024-09-16T03:41:21.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1576999999999997e-05 2024-09-16T03:41:21.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:21.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.486599+0000", 2024-09-16T03:41:21.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0139999999999999e-05 2024-09-16T03:41:21.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:21.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:21.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.493943+0000", 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0073441599999999998 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.493972+0000", 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8853e-05 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.000 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.fs2 97/55 ECSubWrite(tid=409, reqid=osd.0.0:409, at_version=97'103, trim_to=0'0, roll_forward_to=95'99))", 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:16.490534+0000", 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.4100174460000003, 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078940859999999998, 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.490534+0000", 2024-09-16T03:41:22.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.490534+0000", 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.490534+0000", 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.490573+0000", 2024-09-16T03:41:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8794999999999997e-05 2024-09-16T03:41:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.490586+0000", 2024-09-16T03:41:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2594e-05 2024-09-16T03:41:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.490595+0000", 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320000000000002e-06 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.493595+0000", 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030000880000000001 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.493604+0000", 2024-09-16T03:41:22.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4960000000000002e-06 2024-09-16T03:41:22.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.498393+0000", 2024-09-16T03:41:22.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047887959999999997 2024-09-16T03:41:22.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:16.498428+0000", 2024-09-16T03:41:22.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4755999999999997e-05 2024-09-16T03:41:22.006 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.006 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.006 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.006 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.006 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.006 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:22.008 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:41:22.009 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T03:41:22.005+0000 7f7e00a86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33484) UID: 0 2024-09-16T03:41:22.109 INFO:tasks.ceph.osd.11.smithi121.stderr:2024-09-16T03:41:22.108+0000 7fe422eb4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33194) UID: 0 2024-09-16T03:41:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:41:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:22.122 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:41:22.210 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:22.208+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:22.225 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:22.225 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:41:22.225 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:41:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:22.233 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:41:22.310 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-09-16T03:41:22.308+0000 7ff8346a6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33491) UID: 0 2024-09-16T03:41:22.337 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:264 3.7s0 3:e06e137e:::smithi12138290-27:head [copy-from ver 4] snapc 6=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.744537498, 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012313443, 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 264 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590676+0000", 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590679+0000", 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1739999999999999e-06 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590686+0000", 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.43e-06 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590709+0000", 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2748999999999999e-05 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590718+0000", 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0359999999999995e-06 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590860+0000", 2024-09-16T03:41:22.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000142208 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590966+0000", 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000106439 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.599409+0000", 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008442583 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600331+0000", 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000922202 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.602907+0000", 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002575787 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.602990+0000", 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2835000000000002e-05 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.343 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:265 3.7s0 3:e06e137e:::smithi12138290-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.744415658999998, 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0092800720000000003, 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:22.344 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 265 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590798+0000", 2024-09-16T03:41:22.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590799+0000", 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590804+0000", 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3009999999999997e-06 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590832+0000", 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7851000000000002e-05 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.590849+0000", 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6691e-05 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.591224+0000", 2024-09-16T03:41:22.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037502100000000001 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.591256+0000", 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.184e-05 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.599957+0000", 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087011659999999998 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600004+0000", 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7218000000000001e-05 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:21.600078+0000", 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4065000000000003e-05 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.349 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:296 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.864095341000002, 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084845249999999997, 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 296 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471119+0000", 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471120+0000", 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.804e-06 2024-09-16T03:41:22.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471128+0000", 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0199999999999994e-06 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471144+0000", 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5957999999999999e-05 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471153+0000", 2024-09-16T03:41:22.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7299999999999994e-06 2024-09-16T03:41:22.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471238+0000", 2024-09-16T03:41:22.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4860999999999999e-05 2024-09-16T03:41:22.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471244+0000", 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0859999999999998e-06 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.473493+0000", 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022493750000000001 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.473624+0000", 2024-09-16T03:41:22.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130891 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.479603+0000", 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059788000000000003 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:297 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:41:22.355 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.86403894, 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094313820000000003, 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 297 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471175+0000", 2024-09-16T03:41:22.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9900000000000001e-07 2024-09-16T03:41:22.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471180+0000", 2024-09-16T03:41:22.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4619999999999998e-06 2024-09-16T03:41:22.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471186+0000", 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3130000000000004e-06 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471190+0000", 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.194e-06 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471314+0000", 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012333299999999999 2024-09-16T03:41:22.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:22.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471322+0000", 2024-09-16T03:41:22.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8280000000000003e-06 2024-09-16T03:41:22.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.476742+0000", 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054202740000000001 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.476849+0000", 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010663899999999999 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480606+0000", 2024-09-16T03:41:22.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037578400000000001 2024-09-16T03:41:22.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.362 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.362 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:298 3.7s0 3:e441e0cd:::smithi12138290-34:head [read 0~1671168,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:41:22.362 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:41:22.362 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.863990413000003, 2024-09-16T03:41:22.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011872347, 2024-09-16T03:41:22.362 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.363 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.363 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:22.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:22.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:22.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:22.363 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 298 2024-09-16T03:41:22.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.363 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:41:22.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:41:22.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471224+0000", 2024-09-16T03:41:22.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2399999999999999e-07 2024-09-16T03:41:22.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471227+0000", 2024-09-16T03:41:22.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1149999999999998e-06 2024-09-16T03:41:22.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471232+0000", 2024-09-16T03:41:22.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.152e-06 2024-09-16T03:41:22.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471237+0000", 2024-09-16T03:41:22.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9540000000000003e-06 2024-09-16T03:41:22.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471356+0000", 2024-09-16T03:41:22.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011896900000000001 2024-09-16T03:41:22.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:22.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.471364+0000", 2024-09-16T03:41:22.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480706+0000", 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093421570000000002 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.480820+0000", 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011411100000000001 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:26.483096+0000", 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022753819999999998 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:308 3.7s0 3:e441e0cd:::smithi12138290-34:head [create,setxattr h^blbcaNtSmmsQtZPdaRtPkhOOgN_Vi\\f`NtsZ[V (561) in=40b,setxattr T (17332) in=1b,setxattr UVoaZteYfmpthpcfl[gebaR\\TdqcNkhPRRdrgsX] (279) in=40b,setxattr bTPgfXb\\YXtXkQag\\Qh`hlXr[posd]]cje]^eebl (1121) in=40b,setxattr hm^gakWSOOOfNOTph^tma]iaNsaqfjqZYb\\seZNW (3297) in=40b,setxattr YiNn[]m_q`YlQlaV^mRn^ (17332) in=21b,setxattr \\tllpkXThmWXSkOopogrdtjY]f^a^sem_krafjfi (947) in=40b,setxattr RXiRZRQVSaUTaZmQT^bfqZhS^mZsqadaPdW]ijm` (1963) in=40b,setxattr WTmsTZhNpU`tZY\\^aROaNdmtYsjjsgWWboehPOP` (19508) in=40b,setxattr UZmWZeWaoiX]jmbn][UadnNkcZc_hdcjidtNsRYo (1999) in=40b,setxattr NSebjRYYQ`SbaatVjZWNab`cenkljRq`^]Xr\\jRm (11788) in=40b,setxattr ip`daUaop[rdbNPo\\^c[Wcelsl]a][RScQZ[kfa^ (18524) in=40b,setxattr qU_qcNcQpNkegs_Od`jieTXX[mto]lsR^jTcYprR (593) in=40b,setxattr dboVg`srq[bReagdR]fX\\Pe`Warrp]\\[XpSipTN[ (1683) in=40b,setxattr ZdTaVj[ScjsOiQ]a_]R[RQWoYohnjd`fi[dYmfs` (840) in=40b,setxattr N_[blmbpstWhtUgW]ZjZ[\\O_erqcnQN]R_V^U\\RT (1015) in=40b,setxattr jXbP_mNsadnq`UVfmc^lfZUs\\r_[ZnrahqTUk^Zp (383) in=40b,setxattr nmemd\\ipehj\\q^UVY_YtSrhZjcabNXhlai_noQSO (807) in=40b,setxattr Rbc\\cahhlcrckUioieUbQnrseNkZkdPf\\rUmNb]S (384) in=40b,setxattr ZiOkXqg^PobTdZgrtZ]]l^Rap_tfSQoUQqaefj]] (69) in=40b] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.844729485000002, 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013488215, 2024-09-16T03:41:22.369 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 308 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490484+0000", 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490486+0000", 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.601e-06 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490741+0000", 2024-09-16T03:41:22.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00025543300000000001 2024-09-16T03:41:22.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490756+0000", 2024-09-16T03:41:22.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4949e-05 2024-09-16T03:41:22.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.490764+0000", 2024-09-16T03:41:22.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1659999999999997e-06 2024-09-16T03:41:22.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.492684+0000", 2024-09-16T03:41:22.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019205509999999999 2024-09-16T03:41:22.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:22.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.493193+0000", 2024-09-16T03:41:22.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050880499999999998 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.494675+0000", 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001482443 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.496347+0000", 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001671358 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:22.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.503822+0000", 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074755639999999996 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:28.503973+0000", 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000150345 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.9s3 73/55 ECSubWrite(tid=95, reqid=client.4615.0:668, at_version=73'49, trim_to=0'0, roll_forward_to=73'47))", 2024-09-16T03:41:22.375 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.566610+0000", 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.768604282999998, 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0095467829999999997, 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566610+0000", 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566610+0000", 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566610+0000", 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3099999999999997e-07 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566665+0000", 2024-09-16T03:41:22.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4790000000000002e-05 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566678+0000", 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3098999999999999e-05 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566700+0000", 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2004000000000001e-05 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566773+0000", 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3108999999999994e-05 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.566785+0000", 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1325000000000001e-05 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576121+0000", 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093363079999999998 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576156+0000", 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5417000000000003e-05 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.9s3 73/55 ECSubWrite(tid=96, reqid=client.4615.0:669, at_version=73'50, trim_to=0'0, roll_forward_to=73'47))", 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.567667+0000", 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.767546687999999, 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085155620000000008, 2024-09-16T03:41:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567667+0000", 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567667+0000", 2024-09-16T03:41:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567668+0000", 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9999999999999997e-07 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567697+0000", 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8731e-05 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567710+0000", 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3053e-05 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.567718+0000", 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2109999999999998e-06 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568977+0000", 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012587410000000001 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568987+0000", 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0122000000000001e-05 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576161+0000", 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007174003 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.576183+0000", 2024-09-16T03:41:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2001000000000002e-05 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.9s3 73/55 ECSubWrite(tid=97, reqid=client.4615.0:670, at_version=73'51, trim_to=0'0, roll_forward_to=73'47))", 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.568386+0000", 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.766828009000001, 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0097616679999999994, 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.385 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568386+0000", 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568386+0000", 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568387+0000", 2024-09-16T03:41:22.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5000000000000002e-07 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568403+0000", 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6155999999999999e-05 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568421+0000", 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7779999999999999e-05 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.568427+0000", 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0569999999999998e-06 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569278+0000", 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00085171900000000002 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569284+0000", 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578114+0000", 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00883046 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578148+0000", 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3469000000000001e-05 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.389 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4615.0:675 3.7s0 3:e441e0cd:::smithi12138290-34:head [rmxattr NSebjRYYQ`SbaatVjZWNab`cenkljRq`^]Xr\\jRm in=40b,rmxattr RXiRZRQVSaUTaZmQT^bfqZhS^mZsqadaPdW]ijm` in=40b,rmxattr UZmWZeWaoiX]jmbn][UadnNkcZc_hdcjidtNsRYo in=40b,rmxattr ZdTaVj[ScjsOiQ]a_]R[RQWoYohnjd`fi[dYmfs` in=40b,rmxattr bTPgfXb\\YXtXkQag\\Qh`hlXr[posd]]cje]^eebl in=40b,rmxattr dboVg`srq[bReagdR]fX\\Pe`Warrp]\\[XpSipTN[ in=40b,rmxattr h^blbcaNtSmmsQtZPdaRtPkhOOgN_Vi\\f`NtsZ[V in=40b,rmxattr ip`daUaop[rdbNPo\\^c[Wcelsl]a][RScQZ[kfa^ in=40b] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:52.569765+0000", 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.765449390000001, 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087892790000000005, 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-09-16T03:41:22.390 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4615", 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 675 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569765+0000", 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569765+0000", 2024-09-16T03:41:22.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569766+0000", 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.435e-06 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569773+0000", 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8519999999999998e-06 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569784+0000", 2024-09-16T03:41:22.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1236999999999999e-05 2024-09-16T03:41:22.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569790+0000", 2024-09-16T03:41:22.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5210000000000002e-06 2024-09-16T03:41:22.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569882+0000", 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2668000000000001e-05 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.569975+0000", 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2733999999999997e-05 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.570353+0000", 2024-09-16T03:41:22.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037774 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578125+0000", 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077719399999999998 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578181+0000", 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5958999999999997e-05 2024-09-16T03:41:22.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:52.578554+0000", 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037319300000000002 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s4 78/31 ECSubWrite(tid=141, reqid=osd.6.0:141, at_version=78'121, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:57.559137+0000", 2024-09-16T03:41:22.396 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.776076930999999, 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087583680000000007, 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559137+0000", 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559137+0000", 2024-09-16T03:41:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559138+0000", 2024-09-16T03:41:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0900000000000001e-07 2024-09-16T03:41:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559154+0000", 2024-09-16T03:41:22.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6388999999999998e-05 2024-09-16T03:41:22.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559165+0000", 2024-09-16T03:41:22.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0986e-05 2024-09-16T03:41:22.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.559171+0000", 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9270000000000002e-06 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.560571+0000", 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014004079999999999 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.560580+0000", 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7490000000000004e-06 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.567862+0000", 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0072813729999999998 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.567895+0000", 2024-09-16T03:41:22.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3827e-05 2024-09-16T03:41:22.402 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.402 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.402 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.402 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.cs4 78/31 ECSubWrite(tid=236, reqid=osd.0.0:236, at_version=78'124, trim_to=0'0, roll_forward_to=78'120))", 2024-09-16T03:41:22.402 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:40:57.561161+0000", 2024-09-16T03:41:22.402 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.774052783999998, 2024-09-16T03:41:22.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094978739999999999, 2024-09-16T03:41:22.402 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.403 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.403 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.403 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561161+0000", 2024-09-16T03:41:22.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561161+0000", 2024-09-16T03:41:22.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561165+0000", 2024-09-16T03:41:22.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6490000000000001e-06 2024-09-16T03:41:22.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561187+0000", 2024-09-16T03:41:22.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2005999999999999e-05 2024-09-16T03:41:22.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561201+0000", 2024-09-16T03:41:22.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3726e-05 2024-09-16T03:41:22.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561211+0000", 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0088000000000001e-05 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561257+0000", 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6690999999999997e-05 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.561264+0000", 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8179999999999998e-06 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.570617+0000", 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093528239999999992 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:40:57.570659+0000", 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2072e-05 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.407 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.9s3 94/55 ECSubWrite(tid=206, reqid=osd.1.0:206, at_version=94'90, trim_to=0'0, roll_forward_to=91'88))", 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.482104+0000", 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.8531102980000007, 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088140760000000005, 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.482104+0000", 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.482104+0000", 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.482104+0000", 2024-09-16T03:41:22.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.37e-07 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.482133+0000", 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8816e-05 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.482144+0000", 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0540000000000001e-05 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.482153+0000", 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2960000000000004e-06 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.482239+0000", 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6303000000000004e-05 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.482250+0000", 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886e-05 2024-09-16T03:41:22.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.490892+0000", 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086418950000000001 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.490918+0000", 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5803e-05 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.412 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.9s3 94/55 ECSubWrite(tid=208, reqid=osd.1.0:208, at_version=94'92, trim_to=0'0, roll_forward_to=91'88))", 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.484148+0000", 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.851065621, 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010402171999999999, 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484148+0000", 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484148+0000", 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484150+0000", 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.257e-06 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484171+0000", 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1021000000000001e-05 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484182+0000", 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1163e-05 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484190+0000", 2024-09-16T03:41:22.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2160000000000006e-06 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484250+0000", 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0180999999999999e-05 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484259+0000", 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7590000000000003e-06 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.494516+0000", 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010257377 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.494550+0000", 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4198e-05 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.8.23:3 3.7s0 3:e441e0cd:::smithi12138290-34:head [assert-version v65,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.484722+0000", 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.8504922110000006, 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023118998000000002, 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484722+0000", 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484722+0000", 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484723+0000", 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6959999999999999e-06 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484730+0000", 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1310000000000003e-06 2024-09-16T03:41:22.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484741+0000", 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1919000000000001e-05 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.484747+0000", 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8239999999999996e-06 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.490274+0000", 2024-09-16T03:41:22.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055268640000000003 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.490381+0000", 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010637300000000001 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.499792+0000", 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094118670000000008 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.499884+0000", 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1714999999999999e-05 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.507841+0000", 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079566089999999999 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.es5 94/55 ECSubWrite(tid=63, reqid=osd.3.0:63, at_version=94'83, trim_to=0'0, roll_forward_to=91'77))", 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.488556+0000", 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.846658111, 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087061989999999995, 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.488556+0000", 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.488556+0000", 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.488557+0000", 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8999999999999996e-07 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.488576+0000", 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9647e-05 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.488587+0000", 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1069e-05 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.488596+0000", 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9560000000000003e-06 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.490905+0000", 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023086040000000001 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.490913+0000", 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1820000000000006e-06 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.497229+0000", 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006315864 2024-09-16T03:41:22.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.497262+0000", 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3186999999999999e-05 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.cs4 94/88 ECSubWrite(tid=386, reqid=osd.0.0:386, at_version=94'193, trim_to=0'0, roll_forward_to=93'190))", 2024-09-16T03:41:22.427 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.491412+0000", 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.8438022689999993, 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094637530000000001, 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491412+0000", 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491412+0000", 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491413+0000", 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0160000000000001e-06 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491434+0000", 2024-09-16T03:41:22.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1489e-05 2024-09-16T03:41:22.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491449+0000", 2024-09-16T03:41:22.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4912000000000001e-05 2024-09-16T03:41:22.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491457+0000", 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3100000000000001e-06 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491519+0000", 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1943e-05 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.491528+0000", 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8079999999999994e-06 2024-09-16T03:41:22.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.500849+0000", 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093211470000000001 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.500875+0000", 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6128000000000001e-05 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.432 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.1s4 94/31 ECSubWrite(tid=232, reqid=osd.6.0:232, at_version=94'185, trim_to=0'0, roll_forward_to=92'183))", 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.494311+0000", 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.8409028050000007, 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0095132499999999991, 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.494311+0000", 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.494311+0000", 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.494312+0000", 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2200000000000003e-07 2024-09-16T03:41:22.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.494327+0000", 2024-09-16T03:41:22.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:41:22.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.494339+0000", 2024-09-16T03:41:22.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1294999999999999e-05 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.494344+0000", 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.494404+0000", 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0003000000000001e-05 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.494413+0000", 2024-09-16T03:41:22.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6880000000000005e-06 2024-09-16T03:41:22.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.503775+0000", 2024-09-16T03:41:22.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093616020000000001 2024-09-16T03:41:22.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.503824+0000", 2024-09-16T03:41:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9845999999999997e-05 2024-09-16T03:41:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.as4 94/85 ECSubWrite(tid=233, reqid=osd.6.0:233, at_version=94'71, trim_to=0'0, roll_forward_to=91'67))", 2024-09-16T03:41:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.495036+0000", 2024-09-16T03:41:22.439 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.8401783159999994, 2024-09-16T03:41:22.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088316090000000007, 2024-09-16T03:41:22.439 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.439 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.439 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.439 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495036+0000", 2024-09-16T03:41:22.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495036+0000", 2024-09-16T03:41:22.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495036+0000", 2024-09-16T03:41:22.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5300000000000004e-07 2024-09-16T03:41:22.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495054+0000", 2024-09-16T03:41:22.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7377999999999999e-05 2024-09-16T03:41:22.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495062+0000", 2024-09-16T03:41:22.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4400000000000005e-06 2024-09-16T03:41:22.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495069+0000", 2024-09-16T03:41:22.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.759e-06 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495114+0000", 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5627000000000002e-05 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495123+0000", 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1459999999999996e-06 2024-09-16T03:41:22.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.503842+0000", 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087193909999999999 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.503867+0000", 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5215000000000002e-05 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.444 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDECSubOpWrite(3.cs4 94/88 ECSubWrite(tid=388, reqid=osd.0.0:388, at_version=94'195, trim_to=0'0, roll_forward_to=93'190))", 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-09-16T03:41:13.495422+0000", 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.8397922290000004, 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010315896999999999, 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-09-16T03:41:22.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495422+0000", 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495422+0000", 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495422+0000", 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8899999999999999e-07 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495442+0000", 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9451999999999999e-05 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495453+0000", 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1327e-05 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495461+0000", 2024-09-16T03:41:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.447e-06 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495501+0000", 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0521e-05 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.495507+0000", 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4299999999999997e-06 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.505667+0000", 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010159945 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-09-16T03:41:13.505738+0000", 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1086000000000003e-05 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-16T03:41:22.449 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:22.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:41:22.452 INFO:tasks.ceph.osd.14.smithi119.stderr:2024-09-16T03:41:22.449+0000 7eff53556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32992) UID: 0 2024-09-16T03:41:22.534 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'unique_pool_0' removed 2024-09-16T03:41:22.549 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:41:22.552 INFO:tasks.ceph.osd.9.smithi103.stderr:2024-09-16T03:41:22.550+0000 7f8eddab3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33190) UID: 0 2024-09-16T03:41:22.554 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T03:41:22.563 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T03:41:22.571 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:22.571 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:22.571 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:41:22.571 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:22.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:41:22.688 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:22.688 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:22.688 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:41:22.688 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:41:22.688 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:22.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:41:22.804 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:22.804 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:41:22.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:41:22.804 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:41:22.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.804 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=28, reqid=client.4615.0:355, at_version=53'9, trim_to=0'0, roll_forward_to=34'6))", 2024-09-16T03:41:22.804 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:41:22.804 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 50.296508889000002, 2024-09-16T03:41:22.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068498659999999996, 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504747+0000", 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504748+0000", 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4499999999999997e-07 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504768+0000", 2024-09-16T03:41:22.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0582999999999999e-05 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504782+0000", 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3883000000000001e-05 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504790+0000", 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3389999999999993e-06 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504921+0000", 2024-09-16T03:41:22.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013093800000000001 2024-09-16T03:41:22.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.504930+0000", 2024-09-16T03:41:22.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7499999999999992e-06 2024-09-16T03:41:22.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511566+0000", 2024-09-16T03:41:22.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066358449999999996 2024-09-16T03:41:22.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.511597+0000", 2024-09-16T03:41:22.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0883000000000002e-05 2024-09-16T03:41:22.810 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.810 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.810 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 63/31 ECSubWrite(tid=140, reqid=osd.0.0:140, at_version=63'56, trim_to=0'0, roll_forward_to=60'52))", 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.476911+0000", 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 40.324344785999997, 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0063366669999999998, 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476911+0000", 2024-09-16T03:41:22.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476911+0000", 2024-09-16T03:41:22.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476912+0000", 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4099999999999998e-07 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476932+0000", 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0027e-05 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476942+0000", 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7429999999999997e-06 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476950+0000", 2024-09-16T03:41:22.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3510000000000002e-06 2024-09-16T03:41:22.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.478026+0000", 2024-09-16T03:41:22.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001076411 2024-09-16T03:41:22.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.478036+0000", 2024-09-16T03:41:22.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5750000000000007e-06 2024-09-16T03:41:22.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483217+0000", 2024-09-16T03:41:22.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051814000000000001 2024-09-16T03:41:22.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483248+0000", 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0419000000000001e-05 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 63/31 ECSubWrite(tid=31, reqid=osd.15.0:31, at_version=63'24, trim_to=0'0, roll_forward_to=60'22))", 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 40.324321114, 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0062644750000000003, 2024-09-16T03:41:22.816 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:41:22.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476935+0000", 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6899999999999999e-07 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476965+0000", 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9714999999999999e-05 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476988+0000", 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3297000000000001e-05 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.476998+0000", 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0254999999999999e-05 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.477097+0000", 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8939999999999998e-05 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.477108+0000", 2024-09-16T03:41:22.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.039e-05 2024-09-16T03:41:22.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483157+0000", 2024-09-16T03:41:22.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060496719999999999 2024-09-16T03:41:22.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.483199+0000", 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1937000000000002e-05 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.3s2 73/31 ECSubWrite(tid=41, reqid=osd.15.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=73'31))", 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.497839+0000", 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 30.303416472999999, 2024-09-16T03:41:22.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060867669999999999, 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497839+0000", 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497839+0000", 2024-09-16T03:41:22.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497840+0000", 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0100000000000001e-06 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497854+0000", 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3358e-05 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497871+0000", 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6994999999999999e-05 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497880+0000", 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3589999999999997e-06 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497967+0000", 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7106000000000003e-05 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.497978+0000", 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0370999999999999e-05 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.503877+0000", 2024-09-16T03:41:22.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058990930000000002 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.503926+0000", 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9475000000000003e-05 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 74/55 ECSubWrite(tid=30, reqid=osd.3.0:30, at_version=74'47, trim_to=0'0, roll_forward_to=73'39))", 2024-09-16T03:41:22.826 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.535733+0000", 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 29.265523193, 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066050730000000004, 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535733+0000", 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535733+0000", 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535734+0000", 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.246e-06 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535760+0000", 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5706000000000001e-05 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535773+0000", 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3682e-05 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.535782+0000", 2024-09-16T03:41:22.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9090000000000006e-06 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.536271+0000", 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00048888700000000002 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.536282+0000", 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.078e-05 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542291+0000", 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060094379999999998 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542338+0000", 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6424999999999997e-05 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.831 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 74/31 ECSubWrite(tid=131, reqid=osd.12.0:131, at_version=74'68, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.539984+0000", 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 29.261271910000001, 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078357930000000006, 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539984+0000", 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539984+0000", 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539985+0000", 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0299999999999997e-07 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540008+0000", 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2765999999999999e-05 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540022+0000", 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4455e-05 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540031+0000", 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2e-06 2024-09-16T03:41:22.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540256+0000", 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000224391 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540280+0000", 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3952e-05 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547770+0000", 2024-09-16T03:41:22.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007489941 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.547820+0000", 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0185000000000001e-05 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 84/55 ECSubWrite(tid=293, reqid=osd.0.0:293, at_version=84'75, trim_to=0'0, roll_forward_to=83'71))", 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.481368+0000", 2024-09-16T03:41:22.836 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.319888369000001, 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0076366239999999998, 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481368+0000", 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481368+0000", 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481368+0000", 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0299999999999999e-07 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481397+0000", 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8648000000000001e-05 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481406+0000", 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5810000000000003e-06 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481414+0000", 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8499999999999994e-06 2024-09-16T03:41:22.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.482165+0000", 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00075051999999999996 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.482177+0000", 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2602999999999999e-05 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.488963+0000", 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0067854270000000001 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489004+0000", 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1492000000000002e-05 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 84/55 ECSubWrite(tid=50, reqid=osd.3.0:50, at_version=84'71, trim_to=0'0, roll_forward_to=83'69))", 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.483239+0000", 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.318016421999999, 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0064962910000000004, 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483239+0000", 2024-09-16T03:41:22.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483239+0000", 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483240+0000", 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000001e-07 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483267+0000", 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6974e-05 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483284+0000", 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6869999999999999e-05 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483293+0000", 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3400000000000004e-06 2024-09-16T03:41:22.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483382+0000", 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8183000000000006e-05 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.483392+0000", 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0691e-05 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489700+0000", 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0063073249999999999 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489736+0000", 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6229999999999997e-05 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.846 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 84/31 ECSubWrite(tid=88, reqid=osd.8.0:88, at_version=84'69, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.484725+0000", 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.316531168000001, 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060921380000000004, 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484725+0000", 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.847 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484725+0000", 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484725+0000", 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5100000000000001e-07 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484749+0000", 2024-09-16T03:41:22.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3714e-05 2024-09-16T03:41:22.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.849 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.849 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484760+0000", 2024-09-16T03:41:22.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0857e-05 2024-09-16T03:41:22.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.849 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.849 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484768+0000", 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5359999999999992e-06 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484937+0000", 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000169347 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.484946+0000", 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7749999999999998e-06 2024-09-16T03:41:22.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.490785+0000", 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058390350000000002 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.490817+0000", 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1622999999999999e-05 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.851 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.852 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.852 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 84/31 ECSubWrite(tid=89, reqid=osd.8.0:89, at_version=84'71, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T03:41:22.852 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.485262+0000", 2024-09-16T03:41:22.852 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.315993911, 2024-09-16T03:41:22.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010975762, 2024-09-16T03:41:22.852 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.852 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485262+0000", 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485262+0000", 2024-09-16T03:41:22.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485263+0000", 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485286+0000", 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3339000000000001e-05 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485300+0000", 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3658999999999999e-05 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485319+0000", 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8681e-05 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489431+0000", 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041121150000000004 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489441+0000", 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8400000000000007e-06 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496195+0000", 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006754566 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496238+0000", 2024-09-16T03:41:22.856 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2620000000000002e-05 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 84/55 ECSubWrite(tid=294, reqid=osd.0.0:293, at_version=0'0, trim_to=0'0, roll_forward_to=84'75))", 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.489541+0000", 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.311715309, 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0067107429999999999, 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489541+0000", 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489541+0000", 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489542+0000", 2024-09-16T03:41:22.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8100000000000001e-07 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489554+0000", 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2002e-05 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489569+0000", 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5159e-05 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489578+0000", 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3770000000000002e-06 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489633+0000", 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5251999999999999e-05 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489643+0000", 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6190000000000006e-06 2024-09-16T03:41:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496201+0000", 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0065581270000000004 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496251+0000", 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0226000000000003e-05 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.861 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.fs6 84/55 ECSubWrite(tid=295, reqid=osd.0.0:295, at_version=84'77, trim_to=0'0, roll_forward_to=84'75))", 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.491137+0000", 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.310119057000001, 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079076210000000001, 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.862 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491137+0000", 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491137+0000", 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491138+0000", 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0100000000000003e-07 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.863 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491156+0000", 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.838e-05 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491168+0000", 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2320000000000001e-05 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491174+0000", 2024-09-16T03:41:22.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8189999999999997e-06 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491231+0000", 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6564000000000003e-05 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.491243+0000", 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.208e-05 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.499004+0000", 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0077610650000000002 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.499044+0000", 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0491999999999998e-05 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 94/31 ECSubWrite(tid=101, reqid=osd.8.0:101, at_version=94'79, trim_to=0'0, roll_forward_to=91'75))", 2024-09-16T03:41:22.866 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.482418+0000", 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.3188375449999992, 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061555799999999999, 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482418+0000", 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482418+0000", 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482419+0000", 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7000000000000003e-07 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482466+0000", 2024-09-16T03:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6539999999999998e-05 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482479+0000", 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3619e-05 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482492+0000", 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2337999999999999e-05 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482556+0000", 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4647000000000006e-05 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482576+0000", 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9412e-05 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488519+0000", 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059435110000000003 2024-09-16T03:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488574+0000", 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4543000000000003e-05 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 94/55 ECSubWrite(tid=61, reqid=client.4615.0:985, at_version=94'79, trim_to=0'0, roll_forward_to=91'77))", 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.485387+0000", 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.3158693509999999, 2024-09-16T03:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0072071790000000002, 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485387+0000", 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485387+0000", 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485387+0000", 2024-09-16T03:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0999999999999999e-07 2024-09-16T03:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485417+0000", 2024-09-16T03:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0151999999999999e-05 2024-09-16T03:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485427+0000", 2024-09-16T03:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9660000000000008e-06 2024-09-16T03:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485435+0000", 2024-09-16T03:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2879999999999993e-06 2024-09-16T03:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485600+0000", 2024-09-16T03:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000164959 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485611+0000", 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0937999999999999e-05 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.492561+0000", 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069494639999999998 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.492594+0000", 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2901999999999998e-05 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds5 94/85 ECSubWrite(tid=212, reqid=osd.14.0:212, at_version=94'108, trim_to=0'0, roll_forward_to=91'104))", 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.485765+0000", 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.3154911550000001, 2024-09-16T03:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060379489999999999, 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485765+0000", 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485765+0000", 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485766+0000", 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0130000000000001e-06 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485791+0000", 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5695999999999999e-05 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485811+0000", 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9462000000000001e-05 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485821+0000", 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0241000000000001e-05 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485880+0000", 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8477999999999998e-05 2024-09-16T03:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485897+0000", 2024-09-16T03:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7427000000000001e-05 2024-09-16T03:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.491766+0000", 2024-09-16T03:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058688880000000001 2024-09-16T03:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.491803+0000", 2024-09-16T03:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6743999999999999e-05 2024-09-16T03:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 94/55 ECSubWrite(tid=63, reqid=osd.3.0:63, at_version=94'83, trim_to=0'0, roll_forward_to=91'77))", 2024-09-16T03:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.488436+0000", 2024-09-16T03:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.3128201920000002, 2024-09-16T03:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061141540000000001, 2024-09-16T03:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488436+0000", 2024-09-16T03:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488436+0000", 2024-09-16T03:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488437+0000", 2024-09-16T03:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5799999999999998e-07 2024-09-16T03:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488461+0000", 2024-09-16T03:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4178000000000001e-05 2024-09-16T03:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488473+0000", 2024-09-16T03:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1951e-05 2024-09-16T03:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488482+0000", 2024-09-16T03:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9160000000000007e-06 2024-09-16T03:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.490000+0000", 2024-09-16T03:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001518499 2024-09-16T03:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.490009+0000", 2024-09-16T03:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8210000000000008e-06 2024-09-16T03:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.494509+0000", 2024-09-16T03:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044997570000000001 2024-09-16T03:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.494550+0000", 2024-09-16T03:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1174000000000003e-05 2024-09-16T03:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.es4 94/55 ECSubWrite(tid=64, reqid=osd.3.0:63, at_version=0'0, trim_to=0'0, roll_forward_to=94'83))", 2024-09-16T03:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.497633+0000", 2024-09-16T03:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.3036230300000007, 2024-09-16T03:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059829410000000003, 2024-09-16T03:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497633+0000", 2024-09-16T03:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497633+0000", 2024-09-16T03:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.897 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.897 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497634+0000", 2024-09-16T03:41:22.897 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4e-07 2024-09-16T03:41:22.897 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.897 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497649+0000", 2024-09-16T03:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5420000000000001e-05 2024-09-16T03:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497665+0000", 2024-09-16T03:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5764999999999999e-05 2024-09-16T03:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497675+0000", 2024-09-16T03:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6099999999999995e-06 2024-09-16T03:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497740+0000", 2024-09-16T03:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5225e-05 2024-09-16T03:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497750+0000", 2024-09-16T03:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0360000000000001e-05 2024-09-16T03:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.503574+0000", 2024-09-16T03:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058242220000000004 2024-09-16T03:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.503616+0000", 2024-09-16T03:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1399000000000001e-05 2024-09-16T03:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 94/31 ECSubWrite(tid=103, reqid=client.4615.0:982, at_version=94'81, trim_to=0'0, roll_forward_to=94'79))", 2024-09-16T03:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.519271+0000", 2024-09-16T03:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.2819846859999995, 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090919550000000005, 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.519271+0000", 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.519271+0000", 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.519272+0000", 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7700000000000004e-07 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.524471+0000", 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051987270000000002 2024-09-16T03:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.524487+0000", 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5940999999999999e-05 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.524495+0000", 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.49e-06 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.524550+0000", 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5198000000000002e-05 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.524559+0000", 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.528322+0000", 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037636060000000001 2024-09-16T03:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.528363+0000", 2024-09-16T03:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0836e-05 2024-09-16T03:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s5 97/31 ECSubWrite(tid=114, reqid=osd.8.0:114, at_version=97'93, trim_to=0'0, roll_forward_to=97'91))", 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.505013+0000", 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.2962426210000002, 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078207139999999994, 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.505013+0000", 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.505013+0000", 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.505014+0000", 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9599999999999999e-07 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.505032+0000", 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.823e-05 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.505042+0000", 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9140000000000003e-06 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.505049+0000", 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5449999999999998e-06 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.505100+0000", 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0986999999999998e-05 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.505109+0000", 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3100000000000001e-06 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.512800+0000", 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007691537 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.512834+0000", 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3695000000000002e-05 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs5 98/88 ECSubWrite(tid=426, reqid=client.4615.0:1016, at_version=98'228, trim_to=0'0, roll_forward_to=98'227))", 2024-09-16T03:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:17.585328+0000", 2024-09-16T03:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.2159282630000003, 2024-09-16T03:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069741969999999997, 2024-09-16T03:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.585328+0000", 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.585328+0000", 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.585328+0000", 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9999999999999997e-07 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.589826+0000", 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044974969999999996 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.589842+0000", 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6606999999999999e-05 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.589851+0000", 2024-09-16T03:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.85e-06 2024-09-16T03:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.589930+0000", 2024-09-16T03:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8836000000000002e-05 2024-09-16T03:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.589942+0000", 2024-09-16T03:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1629e-05 2024-09-16T03:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.592274+0000", 2024-09-16T03:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023321819999999999 2024-09-16T03:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.592302+0000", 2024-09-16T03:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7895999999999999e-05 2024-09-16T03:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:22.922 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:41:23.027 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:23.027 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:23.027 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:41:23.027 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:23.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:41:23.137 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:23.137 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:23.138 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:41:23.138 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:41:23.138 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:23.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:41:23.249 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:23.249 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:301 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~491520] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 54.763983285999998, 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013883662999999999, 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.250 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 301 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482858+0000", 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.482860+0000", 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.911e-06 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483803+0000", 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00094328700000000001 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483820+0000", 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6249e-05 2024-09-16T03:41:23.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483828+0000", 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.704e-06 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483895+0000", 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6839000000000002e-05 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.483902+0000", 2024-09-16T03:41:23.253 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2169999999999997e-06 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.485797+0000", 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018946049999999999 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.485916+0000", 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011867 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487208+0000", 2024-09-16T03:41:23.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012921110000000001 2024-09-16T03:41:23.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493186+0000", 2024-09-16T03:41:23.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059786010000000001 2024-09-16T03:41:23.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496639+0000", 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034530749999999999 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496742+0000", 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102394 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:302 3.1s0 3:8a735ecd:::smithi12138290-30:head [append 0~770048] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:23.256 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 54.759704012999997, 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014729638999999999, 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 302 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.257 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487137+0000", 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487139+0000", 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.378e-06 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.258 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488116+0000", 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00097674700000000003 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488127+0000", 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1817999999999999e-05 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488135+0000", 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6620000000000005e-06 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489443+0000", 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001308303 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489564+0000", 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000120664 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490525+0000", 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00096115100000000004 2024-09-16T03:41:23.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493256+0000", 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027313429999999998 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501784+0000", 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0085277840000000001 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501867+0000", 2024-09-16T03:41:23.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2788999999999996e-05 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:303 3.1s0 3:8a735ecd:::smithi12138290-30:head [setxattr _header (57) in=7b] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 54.758692476999997, 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013931318, 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.262 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 303 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488149+0000", 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488150+0000", 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000001e-07 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488152+0000", 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.509e-06 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488157+0000", 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5739999999999997e-06 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488160+0000", 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.349e-06 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491536+0000", 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033756799999999998 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491638+0000", 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010186799999999999 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492139+0000", 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050129999999999999 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493257+0000", 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001118551 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.266 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501989+0000", 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087319819999999992 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502080+0000", 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0827000000000003e-05 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.267 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:304 3.1s0 3:8a735ecd:::smithi12138290-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 54.758675498999999, 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.024338029000000001, 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 304 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.269 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488166+0000", 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.53e-07 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488169+0000", 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3309999999999998e-06 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488173+0000", 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:41:23.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.271 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.271 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.488176+0000", 2024-09-16T03:41:23.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0539999999999999e-06 2024-09-16T03:41:23.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.271 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.271 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492435+0000", 2024-09-16T03:41:23.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042594399999999998 2024-09-16T03:41:23.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492534+0000", 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8558000000000001e-05 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505269+0000", 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012735076 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505321+0000", 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1977000000000001e-05 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.512504+0000", 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0071833060000000004 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:309 3.1s0 3:82a78c8d:::smithi12138290-1:head [copy-from ver 4] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:41:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 54.756016045999999, 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020962743999999998, 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 309 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:41:23.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490825+0000", 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490827+0000", 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.275 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490832+0000", 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4550000000000003e-06 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490845+0000", 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2951e-05 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490873+0000", 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7566999999999999e-05 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492571+0000", 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001698359 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492650+0000", 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8858999999999998e-05 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.503971+0000", 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011320805 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511333+0000", 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073624160000000001 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511735+0000", 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00040226999999999998 2024-09-16T03:41:23.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.511788+0000", 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2757999999999999e-05 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:310 3.1s0 3:82a78c8d:::smithi12138290-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 54.755950067000001, 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014337858, 2024-09-16T03:41:23.279 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 310 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490891+0000", 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490892+0000", 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.37e-07 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490896+0000", 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3710000000000002e-06 2024-09-16T03:41:23.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490932+0000", 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5796000000000001e-05 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490941+0000", 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8589999999999994e-06 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493109+0000", 2024-09-16T03:41:23.282 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021679310000000001 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493167+0000", 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7572999999999999e-05 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505077+0000", 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011909968999999999 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.283 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505164+0000", 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7409000000000003e-05 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.505229+0000", 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5213000000000001e-05 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:893 3.1s0 3:82a78c8d:::smithi12138290-1:head [append 0~16384] snapc 1d=[1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:41:23.284 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.506642+0000", 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.74019921, 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.026920999000000001, 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 893 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506642+0000", 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506642+0000", 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506644+0000", 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6169999999999999e-06 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506716+0000", 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2272000000000002e-05 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506731+0000", 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5065999999999999e-05 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506739+0000", 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2230000000000007e-06 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.508774+0000", 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002034264 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.508867+0000", 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3713000000000002e-05 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.509403+0000", 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00053514299999999997 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.512521+0000", 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031181920000000001 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.533480+0000", 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020959177999999998 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.533563+0000", 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3331000000000006e-05 2024-09-16T03:41:23.289 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:894 3.1s0 3:82a78c8d:::smithi12138290-1:head [setxattr _header (56) in=7b] snapc 1d=[1d,1b,1a,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.506763+0000", 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.740078397000001, 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.028329701999999998, 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.290 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 894 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506763+0000", 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506763+0000", 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506764+0000", 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2999999999999999e-07 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506770+0000", 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2219999999999999e-06 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506778+0000", 2024-09-16T03:41:23.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4610000000000002e-06 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506785+0000", 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0790000000000003e-06 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.510080+0000", 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032957709999999999 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.510184+0000", 2024-09-16T03:41:23.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000103644 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.510640+0000", 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000455823 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.515070+0000", 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044299789999999997 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.535029+0000", 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.019959147999999999 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.535093+0000", 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3744999999999994e-05 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:895 3.1s0 3:82a78c8d:::smithi12138290-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:41:23.295 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.506796+0000", 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 19.740045777999999, 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.029474764000000001, 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 895 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.296 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506796+0000", 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506796+0000", 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506796+0000", 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5599999999999995e-07 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506799+0000", 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.497e-06 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506808+0000", 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6379999999999999e-06 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.506815+0000", 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3130000000000004e-06 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.510904+0000", 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040891809999999999 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.510973+0000", 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8979999999999993e-05 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.535419+0000", 2024-09-16T03:41:23.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.024446045999999999 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.535479+0000", 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0438e-05 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.536270+0000", 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000791115 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:970 3.1s0 3:8a735ecd:::smithi12138290-30:head [copy-from ver 34] snapc 24=[24,23,22,21,20,1f,1e,1d,1b,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:11.494650+0000", 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 11.752191536, 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020428584999999999, 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 970 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.301 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494650+0000", 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494650+0000", 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494650+0000", 2024-09-16T03:41:23.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0800000000000001e-07 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494655+0000", 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7310000000000001e-06 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494659+0000", 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.754e-06 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494661+0000", 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.661e-06 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.498097+0000", 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00343583 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.498182+0000", 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4753000000000007e-05 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.506976+0000", 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087937889999999998 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.509015+0000", 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020394739999999999 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.515026+0000", 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060104900000000003 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.515078+0000", 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2794999999999999e-05 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:971 3.1s0 3:8a735ecd:::smithi12138290-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:11.494665+0000", 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 11.752176128, 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014207153, 2024-09-16T03:41:23.306 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 971 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494665+0000", 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494665+0000", 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494666+0000", 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4999999999999999e-07 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494667+0000", 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.381e-06 2024-09-16T03:41:23.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494670+0000", 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3419999999999999e-06 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.494674+0000", 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4489999999999999e-06 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.498334+0000", 2024-09-16T03:41:23.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036606439999999998 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.498378+0000", 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3269000000000002e-05 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.508498+0000", 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010119981 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.508607+0000", 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010894400000000001 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:11.508872+0000", 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026589300000000001 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 94/55 ECSubWrite(tid=206, reqid=osd.1.0:206, at_version=94'90, trim_to=0'0, roll_forward_to=91'88))", 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.482149+0000", 2024-09-16T03:41:23.311 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.7646927960000003, 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017282457000000001, 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482149+0000", 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482149+0000", 2024-09-16T03:41:23.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482149+0000", 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482177+0000", 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8127e-05 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482188+0000", 2024-09-16T03:41:23.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0482000000000001e-05 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.482197+0000", 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.816e-06 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483361+0000", 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001164872 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483373+0000", 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1331999999999999e-05 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.499395+0000", 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016022674000000001 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.499431+0000", 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5593999999999998e-05 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.315 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 94/55 ECSubWrite(tid=208, reqid=osd.1.0:208, at_version=94'92, trim_to=0'0, roll_forward_to=91'88))", 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.483746+0000", 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.7630957760000001, 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015709785, 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483746+0000", 2024-09-16T03:41:23.316 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483746+0000", 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483747+0000", 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9000000000000005e-07 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.317 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483772+0000", 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5097999999999999e-05 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483785+0000", 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3010000000000001e-05 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.483794+0000", 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9309999999999997e-06 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485884+0000", 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020905559999999999 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485903+0000", 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9074000000000001e-05 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.499435+0000", 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013532089000000001 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.499455+0000", 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0037000000000001e-05 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.ds6 94/85 ECSubWrite(tid=212, reqid=osd.14.0:212, at_version=94'108, trim_to=0'0, roll_forward_to=91'104))", 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.485800+0000", 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.7610416400000002, 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012674903, 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.320 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485800+0000", 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485800+0000", 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485801+0000", 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9800000000000002e-07 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485826+0000", 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4898000000000001e-05 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485838+0000", 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2523e-05 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.485860+0000", 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2384000000000001e-05 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.489427+0000", 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0035664410000000001 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.489442+0000", 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4868000000000001e-05 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.323 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.498444+0000", 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090022910000000008 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.498475+0000", 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0599999999999998e-05 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.324 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 97/55 ECSubWrite(tid=219, reqid=osd.1.0:219, at_version=97'98, trim_to=0'0, roll_forward_to=95'96))", 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.481602+0000", 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.7652397889999998, 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013086326000000001, 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.481602+0000", 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.481602+0000", 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.481602+0000", 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0100000000000001e-07 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.481614+0000", 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2279000000000001e-05 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482579+0000", 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00096509299999999998 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482590+0000", 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0523000000000001e-05 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482724+0000", 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013399500000000001 2024-09-16T03:41:23.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482733+0000", 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1929999999999999e-06 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.494661+0000", 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011928533999999999 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.494688+0000", 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6508000000000001e-05 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 97/55 ECSubWrite(tid=221, reqid=client.4615.0:995, at_version=97'100, trim_to=0'0, roll_forward_to=95'96))", 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.484613+0000", 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.7622282279999997, 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017659327999999998, 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.329 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484613+0000", 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484613+0000", 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484614+0000", 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0999999999999998e-07 2024-09-16T03:41:23.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484633+0000", 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9366999999999999e-05 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484652+0000", 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9177999999999998e-05 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484660+0000", 2024-09-16T03:41:23.331 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7729999999999999e-06 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.486785+0000", 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002125002 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.486795+0000", 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9450000000000005e-06 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.332 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502244+0000", 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015448638000000001 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502273+0000", 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8714999999999999e-05 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 97/55 ECSubWrite(tid=222, reqid=client.4615.0:996, at_version=97'101, trim_to=0'0, roll_forward_to=95'96))", 2024-09-16T03:41:23.333 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.484683+0000", 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.7621584099999996, 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017620747999999999, 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484683+0000", 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484683+0000", 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484683+0000", 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0699999999999998e-07 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484698+0000", 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4990999999999999e-05 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484706+0000", 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7020000000000001e-06 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484712+0000", 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3749999999999999e-06 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.491681+0000", 2024-09-16T03:41:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006968602 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.491689+0000", 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502278+0000", 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010589215000000001 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502304+0000", 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5372999999999999e-05 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s6 97/55 ECSubWrite(tid=223, reqid=client.4615.0:997, at_version=97'102, trim_to=0'0, roll_forward_to=95'96))", 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.487124+0000", 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.75971777, 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015202571, 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487124+0000", 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487124+0000", 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487124+0000", 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.51e-07 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487141+0000", 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.717e-05 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487151+0000", 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.012e-05 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487160+0000", 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0539999999999995e-06 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.496120+0000", 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0089605789999999998 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.496130+0000", 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4089999999999992e-06 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502306+0000", 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061766759999999999 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.502326+0000", 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0012000000000001e-05 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.342 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:1020 3.1s0 3:8df00818:::smithi12138290-14:head [rollback 23] snapc 27=[27,26,25,24,23,22,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:18.512092+0000", 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.7347494140000004, 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012987774000000001, 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1020 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.343 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512092+0000", 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512092+0000", 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512092+0000", 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3299999999999998e-07 2024-09-16T03:41:23.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512095+0000", 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2900000000000001e-06 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512099+0000", 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2139999999999998e-06 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512102+0000", 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0809999999999998e-06 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512198+0000", 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6526000000000004e-05 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512204+0000", 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3059999999999997e-06 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.518864+0000", 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066604489999999997 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.518927+0000", 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3215999999999999e-05 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.519287+0000", 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00035971700000000001 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.520343+0000", 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001056345 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.525035+0000", 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046914419999999997 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.525080+0000", 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4855000000000003e-05 2024-09-16T03:41:23.348 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:1021 3.1s0 3:8df00818:::smithi12138290-14:head [append 0~0] snapc 27=[27,26,25,24,23,22,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:18.512120+0000", 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.7347215299999998, 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013078976000000001, 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.349 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1021 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512120+0000", 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512120+0000", 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512120+0000", 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8099999999999999e-07 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512122+0000", 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.641e-06 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512126+0000", 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1409999999999998e-06 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512131+0000", 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7820000000000001e-06 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512232+0000", 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010084100000000001 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.512235+0000", 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3060000000000002e-06 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.519758+0000", 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075227749999999998 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.519822+0000", 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4108000000000004e-05 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.520137+0000", 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00031517900000000001 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.520566+0000", 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00042865299999999998 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.525149+0000", 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045838559999999999 2024-09-16T03:41:23.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:18.525199+0000", 2024-09-16T03:41:23.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9413000000000003e-05 2024-09-16T03:41:23.355 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.355 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.355 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.355 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.355 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.355 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:23.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:41:23.459 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:23.459 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:23.460 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:41:23.460 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:23.468 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:41:23.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T03:41:23.576 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:23.576 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:23.576 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:41:23.576 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:41:23.576 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:23.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:41:23.636 INFO:teuthology.orchestra.run.smithi076.stderr:no valid command found; 10 closest matches: 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:0 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:1 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:2 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:abort 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:assert 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:bluefs files list 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:bluefs stats 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:bluestore allocator dump block 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:bluestore allocator fragmentation block 2024-09-16T03:41:23.637 INFO:teuthology.orchestra.run.smithi076.stderr:admin_socket: invalid command 2024-09-16T03:41:23.639 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:41:23.639 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['dump_ops_in_flight'] 2024-09-16T03:41:23.689 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:23.689 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:41:23.689 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:41:23.689 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:41:23.689 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.689 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:282 3.4s0 3:2307c0f5:::smithi12138290-28:head [append 0~458752] snapc 7=[7,5,3] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 61.111782433999998, 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0093867299999999994, 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 282 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574791+0000", 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.574793+0000", 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5799999999999999e-06 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575536+0000", 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00074309099999999998 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575567+0000", 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0382000000000002e-05 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575590+0000", 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3215000000000001e-05 2024-09-16T03:41:23.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575665+0000", 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5462000000000004e-05 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.575671+0000", 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6969999999999998e-06 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577693+0000", 2024-09-16T03:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020222439999999999 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.577796+0000", 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010306000000000001 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.578998+0000", 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012013880000000001 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.583553+0000", 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045549340000000001 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584072+0000", 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00051922199999999996 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:22.584177+0000", 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105455 2024-09-16T03:41:23.695 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:311 3.bs0 3:db21b75b:::smithi12138290-3:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 55.195630332, 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013432349, 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.696 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 311 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490943+0000", 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490945+0000", 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.419e-06 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490953+0000", 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7379999999999994e-06 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490973+0000", 2024-09-16T03:41:23.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0460999999999999e-05 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490984+0000", 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0857e-05 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491053+0000", 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8351999999999999e-05 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491061+0000", 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3569999999999998e-06 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493588+0000", 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025267430000000001 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.493683+0000", 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4876000000000004e-05 2024-09-16T03:41:23.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496393+0000", 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027106500000000002 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504203+0000", 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078100080000000002 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504276+0000", 2024-09-16T03:41:23.701 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2601000000000005e-05 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504375+0000", 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9287e-05 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:312 3.bs0 3:db21b75b:::smithi12138290-3:head [rollback a] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:41:23.702 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 55.195564707000003, 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013925697000000001, 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 312 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:41:23.703 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491008+0000", 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491009+0000", 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0200000000000001e-07 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491013+0000", 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8160000000000004e-06 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491022+0000", 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6649999999999992e-06 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491028+0000", 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4760000000000003e-06 2024-09-16T03:41:23.705 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491124+0000", 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6192000000000003e-05 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491132+0000", 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6539999999999999e-06 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.706 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499113+0000", 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079811880000000002 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499184+0000", 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0749999999999999e-05 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.499557+0000", 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037325200000000002 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.707 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504380+0000", 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048224590000000003 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504852+0000", 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047194899999999999 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504934+0000", 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2594000000000002e-05 2024-09-16T03:41:23.708 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:313 3.bs0 3:db21b75b:::smithi12138290-3:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 55.195523066, 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016369406999999999, 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.709 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 313 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.710 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491050+0000", 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491051+0000", 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.82e-07 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491054+0000", 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5260000000000002e-06 2024-09-16T03:41:23.711 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491063+0000", 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.354e-06 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491069+0000", 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2299999999999996e-06 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.712 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491177+0000", 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000108285 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.491184+0000", 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0489999999999998e-06 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.501958+0000", 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010773448 2024-09-16T03:41:23.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502064+0000", 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106661 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.502545+0000", 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000481077 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.504381+0000", 2024-09-16T03:41:23.714 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018352970000000001 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.507349+0000", 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002968666 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.507420+0000", 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0232000000000001e-05 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.715 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:371 3.4s0 3:2ef36fe6:::smithi12138290-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 51.156883995000001, 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0095505, 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 371 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.716 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529689+0000", 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7799999999999997e-07 2024-09-16T03:41:23.717 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529691+0000", 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2330000000000001e-06 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529694+0000", 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5489999999999998e-06 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.529697+0000", 2024-09-16T03:41:23.718 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.706e-06 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533902+0000", 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004204564 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.533982+0000", 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0399000000000001e-05 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.719 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.539228+0000", 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052458579999999999 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.539240+0000", 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1912999999999999e-05 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(osd.11.23:1 3.bs0 3:de0fee80:::smithi12138290-48:head [assert-version v28,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-16T03:41:23.720 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:42.479910+0000", 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 41.206662706000003, 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092349349999999997, 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479910+0000", 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479910+0000", 2024-09-16T03:41:23.721 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479913+0000", 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0600000000000002e-06 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479920+0000", 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6290000000000001e-06 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.722 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479936+0000", 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5749000000000001e-05 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.479945+0000", 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4329999999999993e-06 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480018+0000", 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2384000000000004e-05 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.723 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.480125+0000", 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107074 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:42.489145+0000", 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090206060000000005 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.724 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 73/31 ECSubWrite(tid=176, reqid=osd.0.0:176, at_version=73'82, trim_to=0'0, roll_forward_to=72'78))", 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.481052+0000", 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 31.205521016999999, 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012524485, 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481052+0000", 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.725 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481052+0000", 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481053+0000", 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9300000000000006e-07 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481077+0000", 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4111999999999999e-05 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.726 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481098+0000", 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0704e-05 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481107+0000", 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8820000000000007e-06 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481195+0000", 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7662000000000002e-05 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.727 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.481204+0000", 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3980000000000005e-06 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493544+0000", 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012339902 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493577+0000", 2024-09-16T03:41:23.728 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2832e-05 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:649 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~425984] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.483387+0000", 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 31.203186015, 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.021278157999999998, 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.729 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 649 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.483387+0000", 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.483387+0000", 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.730 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.483389+0000", 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1600000000000001e-06 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.484029+0000", 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00063944299999999996 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.484041+0000", 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2369e-05 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.731 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.484049+0000", 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9619999999999993e-06 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.486874+0000", 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028244390000000002 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487037+0000", 2024-09-16T03:41:23.732 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016383700000000001 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.488288+0000", 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001250995 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493564+0000", 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052756249999999999 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504598+0000", 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011034089 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504665+0000", 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7238999999999998e-05 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:650 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~606208] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:23.734 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.487370+0000", 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 31.199203674, 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017507029, 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 650 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487370+0000", 2024-09-16T03:41:23.735 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487370+0000", 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487371+0000", 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9570000000000001e-06 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.736 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487965+0000", 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00059316399999999997 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487986+0000", 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1182000000000001e-05 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.487996+0000", 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0146000000000001e-05 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.737 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.490298+0000", 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023018779999999998 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.490410+0000", 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000112457 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492647+0000", 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002236465 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.738 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499318+0000", 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066710809999999997 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504802+0000", 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054842770000000001 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504877+0000", 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4422e-05 2024-09-16T03:41:23.739 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:651 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~540672] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.491437+0000", 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 31.195136653999999, 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013890236, 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.740 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 651 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491437+0000", 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491437+0000", 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.491438+0000", 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.578e-06 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492172+0000", 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073402799999999996 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492186+0000", 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4137e-05 2024-09-16T03:41:23.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.492195+0000", 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2619999999999998e-06 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493674+0000", 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001479803 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.493779+0000", 2024-09-16T03:41:23.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000104399 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.494705+0000", 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00092621299999999995 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499319+0000", 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046143490000000002 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505255+0000", 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059355179999999999 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505327+0000", 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1948999999999998e-05 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:652 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~409600] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:23.745 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.495027+0000", 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 31.191545848000001, 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010815300999999999, 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 652 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.746 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495027+0000", 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495027+0000", 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495029+0000", 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0789999999999999e-06 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495441+0000", 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000411753 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495457+0000", 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495466+0000", 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9279999999999999e-06 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495851+0000", 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038526000000000001 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496004+0000", 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015289200000000001 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.496867+0000", 2024-09-16T03:41:23.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00086269600000000004 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504685+0000", 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078182930000000005 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505774+0000", 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010881599999999999 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.505843+0000", 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9101000000000001e-05 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:653 3.bs0 3:de0fee80:::smithi12138290-48:head [setxattr _header (57) in=7b] snapc 19=[19,18,16,13,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 31.191090646999999, 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013098884, 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.751 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 653 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495483+0000", 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4600000000000003e-07 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495487+0000", 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9389999999999999e-06 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495495+0000", 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4100000000000002e-06 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495500+0000", 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6300000000000003e-06 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499810+0000", 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043094860000000004 2024-09-16T03:41:23.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.499920+0000", 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000110507 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.500346+0000", 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000425735 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.504687+0000", 2024-09-16T03:41:23.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043411109999999999 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508462+0000", 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037750840000000002 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508581+0000", 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011913600000000001 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:654 3.bs0 3:de0fee80:::smithi12138290-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 31.191063590999999, 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016989471999999999, 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 654 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.757 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495510+0000", 2024-09-16T03:41:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3900000000000005e-07 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495512+0000", 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2259999999999999e-06 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495519+0000", 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7769999999999997e-06 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.495525+0000", 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4419999999999997e-06 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.500653+0000", 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051282619999999998 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.500758+0000", 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105141 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508620+0000", 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078621590000000005 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.508703+0000", 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2622999999999998e-05 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:52.512499+0000", 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003796303 2024-09-16T03:41:23.761 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 74/31 ECSubWrite(tid=194, reqid=osd.0.0:194, at_version=74'100, trim_to=0'0, roll_forward_to=73'96))", 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.530401+0000", 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 30.15617185, 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092564819999999999, 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530401+0000", 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530401+0000", 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530402+0000", 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2900000000000004e-07 2024-09-16T03:41:23.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530430+0000", 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7979000000000001e-05 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530441+0000", 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0855e-05 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530449+0000", 2024-09-16T03:41:23.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1750000000000005e-06 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530878+0000", 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00042890400000000003 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.530941+0000", 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3035999999999998e-05 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539613+0000", 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086722910000000004 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539658+0000", 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4712999999999997e-05 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:790 3.bs0 3:de0fee80:::smithi12138290-48:head [append 0~0] snapc 1b=[1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:57.545852+0000", 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 26.140721120999999, 2024-09-16T03:41:23.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0097149109999999997, 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 790 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545852+0000", 2024-09-16T03:41:23.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545852+0000", 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545852+0000", 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.32e-07 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.768 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545857+0000", 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4410000000000003e-06 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545863+0000", 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2729999999999999e-06 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.545869+0000", 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3739999999999997e-06 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.548747+0000", 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028787299999999999 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.548802+0000", 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4846000000000003e-05 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.549282+0000", 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047939 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.552414+0000", 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031319659999999999 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.555487+0000", 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030735099999999998 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:57.555567+0000", 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9949000000000003e-05 2024-09-16T03:41:23.771 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.5s4 84/31 ECSubWrite(tid=89, reqid=osd.8.0:89, at_version=84'71, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.485270+0000", 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 20.20130326, 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010818193, 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485270+0000", 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485270+0000", 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485271+0000", 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000001e-07 2024-09-16T03:41:23.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485288+0000", 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7501000000000001e-05 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485304+0000", 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6279000000000001e-05 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485313+0000", 2024-09-16T03:41:23.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1920000000000005e-06 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485372+0000", 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9685000000000002e-05 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.485382+0000", 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4539999999999996e-06 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496052+0000", 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010669931000000001 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.496088+0000", 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6473000000000001e-05 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.8s1 84/31 ECSubWrite(tid=220, reqid=osd.11.0:220, at_version=84'33, trim_to=0'0, roll_forward_to=83'31))", 2024-09-16T03:41:23.776 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.489139+0000", 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 20.197434640000001, 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0097139679999999999, 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489139+0000", 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489139+0000", 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489139+0000", 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2200000000000003e-07 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489162+0000", 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3059999999999999e-05 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489177+0000", 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4408e-05 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489185+0000", 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5969999999999992e-06 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489252+0000", 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6661000000000004e-05 2024-09-16T03:41:23.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489262+0000", 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0373999999999999e-05 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.498811+0000", 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.009548974 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.498853+0000", 2024-09-16T03:41:23.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1171999999999998e-05 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.cs1 97/88 ECSubWrite(tid=408, reqid=osd.0.0:408, at_version=97'213, trim_to=0'0, roll_forward_to=95'209))", 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.490360+0000", 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.1962128849999996, 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010242849, 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490360+0000", 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490360+0000", 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490361+0000", 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5199999999999996e-07 2024-09-16T03:41:23.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490380+0000", 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8499999999999999e-05 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490394+0000", 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4338999999999999e-05 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490449+0000", 2024-09-16T03:41:23.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4824999999999998e-05 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490733+0000", 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00028452600000000001 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.490742+0000", 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.867e-06 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.500570+0000", 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0098277280000000009 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.500603+0000", 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3312000000000002e-05 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:1006 3.4s0 3:2307c0f5:::smithi12138290-28:head [setxattr _header (57) in=7b] snapc 26=[26,25,24,23,22,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:17.494675+0000", 2024-09-16T03:41:23.785 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.1918982839999996, 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0098785980000000006, 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1006 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494675+0000", 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494675+0000", 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494676+0000", 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9500000000000002e-07 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494679+0000", 2024-09-16T03:41:23.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0900000000000001e-06 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494684+0000", 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8000000000000004e-06 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494690+0000", 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4949999999999999e-06 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.496608+0000", 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001918182 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.496718+0000", 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000109615 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.497178+0000", 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00046017999999999997 2024-09-16T03:41:23.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.504069+0000", 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068907100000000004 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.504157+0000", 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8292000000000001e-05 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.504553+0000", 2024-09-16T03:41:23.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00039653900000000001 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:1007 3.4s0 3:2307c0f5:::smithi12138290-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:17.494700+0000", 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 6.19187332, 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010882404, 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:23.791 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1007 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494700+0000", 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494700+0000", 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:23.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494700+0000", 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.75e-07 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494703+0000", 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7159999999999999e-06 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494710+0000", 2024-09-16T03:41:23.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8349999999999998e-06 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.494715+0000", 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5430000000000001e-06 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.497478+0000", 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027628079999999998 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:23.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.497567+0000", 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9209000000000006e-05 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.504607+0000", 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0070398880000000002 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.504700+0000", 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2746999999999998e-05 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:23.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:23.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:17.505582+0000", 2024-09-16T03:41:23.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00088218300000000003 2024-09-16T03:41:23.796 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.796 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.796 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.796 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:23.796 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:23.796 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:23.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:41:23.900 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:23.900 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:23.900 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-16T03:41:23.901 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:23.909 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:41:24.011 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:24.011 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-16T03:41:24.011 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-16T03:41:24.012 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-16T03:41:24.012 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:24.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-16T03:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-16T03:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-16T03:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-16T03:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:205 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~655360] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 67.598219416999996, 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075538640000000004, 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 205 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523023+0000", 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523025+0000", 2024-09-16T03:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3519999999999999e-06 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523819+0000", 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00079422800000000001 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523834+0000", 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5333e-05 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523842+0000", 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2960000000000001e-06 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523903+0000", 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1124000000000006e-05 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.523981+0000", 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8166999999999999e-05 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.525035+0000", 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010540969999999999 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.528679+0000", 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036443090000000001 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530521+0000", 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018416789999999999 2024-09-16T03:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:16.530577+0000", 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6279000000000001e-05 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:305 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 55.633993683, 2024-09-16T03:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087742589999999995, 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 305 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487249+0000", 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487251+0000", 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.824e-06 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487259+0000", 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1689999999999992e-06 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487275+0000", 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5889e-05 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487283+0000", 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9710000000000004e-06 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487385+0000", 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010225099999999999 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487554+0000", 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016910199999999999 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489000+0000", 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001445499 2024-09-16T03:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492361+0000", 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033615799999999999 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.495935+0000", 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0035732899999999998 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.496023+0000", 2024-09-16T03:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8684000000000007e-05 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:306 3.6s0 3:64535dd7:::smithi12138290-18:head [rollback 5] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 55.633940821000003, 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0099232220000000006, 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 306 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487302+0000", 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487303+0000", 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.44e-07 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487306+0000", 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.817e-06 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487314+0000", 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4000000000000003e-06 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487319+0000", 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2379999999999997e-06 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489535+0000", 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002216347 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.489617+0000", 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1559999999999998e-05 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490002+0000", 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038484000000000001 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492363+0000", 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023609070000000002 2024-09-16T03:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497154+0000", 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047910110000000004 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497225+0000", 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1557999999999995e-05 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:307 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc c=[c,b,a,9,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 55.633899005000004, 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010112101, 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 307 2024-09-16T03:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487344+0000", 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.51e-07 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487356+0000", 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1793999999999999e-05 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487362+0000", 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8100000000000003e-06 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.487366+0000", 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8670000000000004e-06 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490302+0000", 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029359299999999998 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490396+0000", 2024-09-16T03:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4134000000000004e-05 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.490929+0000", 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00053311500000000002 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.492364+0000", 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014347279999999999 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497383+0000", 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050189220000000003 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:28.497456+0000", 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3250000000000005e-05 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:349 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [rollback 3] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 51.622944468999997, 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084181240000000008, 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 349 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498298+0000", 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498299+0000", 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6600000000000002e-07 2024-09-16T03:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498301+0000", 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3860000000000001e-06 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498307+0000", 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8590000000000001e-06 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498312+0000", 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6249999999999998e-06 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499617+0000", 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0013049660000000001 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.499672+0000", 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5787e-05 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500118+0000", 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044587499999999998 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505378+0000", 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005259789 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506634+0000", 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012560169999999999 2024-09-16T03:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506716+0000", 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2354000000000005e-05 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:350 3.6s0 3:6a9a2fa6:::smithi12138290-19:head [append 0~0] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 51.622924396999998, 2024-09-16T03:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086865829999999995, 2024-09-16T03:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 350 2024-09-16T03:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498318+0000", 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498319+0000", 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.39e-07 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498321+0000", 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.815e-06 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498325+0000", 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8749999999999999e-06 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.498329+0000", 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.014e-06 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500398+0000", 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020690069999999999 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500458+0000", 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9429999999999999e-05 2024-09-16T03:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.500725+0000", 2024-09-16T03:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026708899999999998 2024-09-16T03:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.505380+0000", 2024-09-16T03:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046551040000000002 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.506933+0000", 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015530769999999999 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.507005+0000", 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1833000000000002e-05 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.9s2 53/31 ECSubWrite(tid=40, reqid=osd.13.0:40, at_version=53'20, trim_to=0'0, roll_forward_to=52'18))", 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 51.619657064999998, 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007493184, 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501586+0000", 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.23e-07 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501604+0000", 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8465e-05 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501615+0000", 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.06e-05 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501623+0000", 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8019999999999992e-06 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501779+0000", 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000156093 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.501788+0000", 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3840000000000003e-06 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.509033+0000", 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0072446300000000002 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.509079+0000", 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5986999999999999e-05 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:412 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~737280] snapc d=[d,b,8,5,3] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 51.518311658000002, 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007659028, 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 412 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602931+0000", 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.602932+0000", 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3499999999999995e-07 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603668+0000", 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073570899999999997 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603677+0000", 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4180000000000003e-06 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603702+0000", 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5225999999999999e-05 2024-09-16T03:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603754+0000", 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1391000000000003e-05 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.603826+0000", 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2648e-05 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.604587+0000", 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076084500000000001 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.608450+0000", 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038632409999999999 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610532+0000", 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020811580000000001 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:32.610590+0000", 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8656999999999998e-05 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:466 3.6s0 3:6167ac73:::smithi12138290-7:head [stat,delete] snapc 11=[11,10,f,e,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 45.646198310999999, 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0074852260000000002, 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 466 2024-09-16T03:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475045+0000", 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475046+0000", 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3349999999999999e-06 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475057+0000", 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1158999999999999e-05 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475069+0000", 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1640999999999999e-05 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475076+0000", 2024-09-16T03:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5379999999999997e-06 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475130+0000", 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4267000000000001e-05 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.475135+0000", 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8640000000000004e-06 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478234+0000", 2024-09-16T03:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030989450000000001 2024-09-16T03:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478358+0000", 2024-09-16T03:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000123666 2024-09-16T03:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.478978+0000", 2024-09-16T03:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00061992000000000004 2024-09-16T03:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.479700+0000", 2024-09-16T03:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00072176900000000003 2024-09-16T03:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.482485+0000", 2024-09-16T03:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027855340000000001 2024-09-16T03:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:38.482530+0000", 2024-09-16T03:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4588000000000001e-05 2024-09-16T03:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:688 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~475136] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.539837+0000", 2024-09-16T03:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 30.581406196, 2024-09-16T03:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010608545, 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 688 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539837+0000", 2024-09-16T03:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539837+0000", 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.539839+0000", 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2479999999999999e-06 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540778+0000", 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00093899100000000002 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540797+0000", 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9191e-05 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540806+0000", 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6200000000000005e-06 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.540874+0000", 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7949999999999998e-05 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.541002+0000", 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000128131 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.542267+0000", 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012655019999999999 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550290+0000", 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080230189999999993 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550352+0000", 2024-09-16T03:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2162999999999994e-05 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550445+0000", 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2730000000000002e-05 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:689 3.6s0 3:656a4357:::smithi12138290-23:head [append 0~655360] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:53.544092+0000", 2024-09-16T03:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 30.577150663000001, 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081954690000000004, 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 689 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544092+0000", 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544092+0000", 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.544094+0000", 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8619999999999999e-06 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545122+0000", 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010280389999999999 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545135+0000", 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3426e-05 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545143+0000", 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8269999999999998e-06 2024-09-16T03:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545656+0000", 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00051290899999999998 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.545817+0000", 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016028200000000001 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.546825+0000", 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001008253 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.550450+0000", 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036248209999999999 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.552236+0000", 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017865380000000001 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:53.552288+0000", 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1511999999999997e-05 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:821 3.6s0 3:616ac6f8:::smithi12138290-8:head [append 0~0] snapc 1c=[1c,1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:58.596286+0000", 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.524957194999999, 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080138029999999999, 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 821 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596286+0000", 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596286+0000", 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596287+0000", 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.699e-06 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596298+0000", 2024-09-16T03:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1104000000000001e-05 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596316+0000", 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7652999999999999e-05 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596324+0000", 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0690000000000002e-06 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596454+0000", 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000129863 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596460+0000", 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9540000000000001e-06 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.598917+0000", 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024574169999999999 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.599031+0000", 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011393800000000001 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.599896+0000", 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000864319 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.602014+0000", 2024-09-16T03:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021184149999999998 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.604224+0000", 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022102519999999998 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.604299+0000", 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5119999999999999e-05 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:822 3.6s0 3:616ac6f8:::smithi12138290-8:head [rollback 11] snapc 1c=[1c,1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:58.596341+0000", 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.524902037, 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011007497999999999, 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 822 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596341+0000", 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596341+0000", 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596341+0000", 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6199999999999999e-07 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596343+0000", 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9e-06 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596348+0000", 2024-09-16T03:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596351+0000", 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.202e-06 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596536+0000", 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000184926 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596541+0000", 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.318e-06 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.600433+0000", 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038927150000000002 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.600496+0000", 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.600857+0000", 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00036071600000000002 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.602040+0000", 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001182673 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.607301+0000", 2024-09-16T03:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052608639999999996 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.607348+0000", 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7763000000000003e-05 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:823 3.6s0 3:616ac6f8:::smithi12138290-8:head [append 0~0] snapc 1c=[1c,1b,1a,19,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:40:58.596421+0000", 2024-09-16T03:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 25.524822030999999, 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011103531999999999, 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 823 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596421+0000", 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596421+0000", 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596421+0000", 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.84e-07 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596423+0000", 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0490000000000002e-06 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596428+0000", 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3129999999999998e-06 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596432+0000", 2024-09-16T03:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.196e-06 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596569+0000", 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000137121 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.596572+0000", 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.354e-06 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.601159+0000", 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004586789 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.601261+0000", 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010241799999999999 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.601726+0000", 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00046440199999999999 2024-09-16T03:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.602186+0000", 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00046048399999999999 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.607457+0000", 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052708670000000003 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:40:58.607524+0000", 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7255000000000006e-05 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:892 3.6s0 3:656a4357:::smithi12138290-23:head [delete] snapc 1d=[1d,1b,1a,11,b,8] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.480742+0000", 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 20.640500736, 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014033755, 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 892 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480742+0000", 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480742+0000", 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480744+0000", 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3439999999999999e-06 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480753+0000", 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0760000000000003e-06 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480769+0000", 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6079999999999999e-05 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.480778+0000", 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4129999999999995e-06 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.482114+0000", 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0013362179999999999 2024-09-16T03:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.482254+0000", 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000140056 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.494776+0000", 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012521568 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.bs6 84/31 ECSubWrite(tid=137, reqid=osd.10.0:137, at_version=84'66, trim_to=0'0, roll_forward_to=82'64))", 2024-09-16T03:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:03.481690+0000", 2024-09-16T03:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 20.639552782999999, 2024-09-16T03:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075420139999999997, 2024-09-16T03:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481690+0000", 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481690+0000", 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481690+0000", 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5900000000000003e-07 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481783+0000", 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2295999999999999e-05 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481793+0000", 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0363000000000001e-05 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481801+0000", 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1880000000000002e-06 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481862+0000", 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0949000000000002e-05 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.481873+0000", 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0847e-05 2024-09-16T03:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489183+0000", 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007309505 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:03.489232+0000", 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9506999999999999e-05 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDECSubOpWrite(3.bs6 94/31 ECSubWrite(tid=161, reqid=osd.10.0:161, at_version=94'76, trim_to=0'0, roll_forward_to=91'72))", 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:13.488968+0000", 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 10.632275308000001, 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0088018090000000007, 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488968+0000", 2024-09-16T03:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488968+0000", 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488968+0000", 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1799999999999995e-07 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488989+0000", 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1359999999999999e-05 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.488998+0000", 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9379999999999998e-06 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.489006+0000", 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5549999999999997e-06 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.489107+0000", 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000101056 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.489123+0000", 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5637e-05 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497728+0000", 2024-09-16T03:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086051900000000004 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:13.497769+0000", 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1554999999999998e-05 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:998 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc 26=[26,25,24,23,22,21,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.482625+0000", 2024-09-16T03:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.6386180689999996, 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.01249548, 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 998 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482625+0000", 2024-09-16T03:41:24.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482625+0000", 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482627+0000", 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8309999999999999e-06 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482632+0000", 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.84e-06 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482646+0000", 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3271999999999999e-05 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482653+0000", 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.391e-06 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482713+0000", 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9732999999999999e-05 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482721+0000", 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2109999999999998e-06 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484365+0000", 2024-09-16T03:41:24.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001643512 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.484488+0000", 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000123516 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.485295+0000", 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000807386 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.230 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.494614+0000", 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0093186829999999995 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.495033+0000", 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00041857299999999999 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.495120+0000", 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7532000000000002e-05 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.231 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:999 3.6s0 3:64535dd7:::smithi12138290-18:head [rollback 1d] snapc 26=[26,25,24,23,22,21,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.482760+0000", 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.6384829429999996, 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.018206513000000001, 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.232 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 999 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482760+0000", 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482760+0000", 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482761+0000", 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2399999999999998e-07 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482764+0000", 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.439e-06 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482771+0000", 2024-09-16T03:41:24.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3919999999999997e-06 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482777+0000", 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1e-06 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482822+0000", 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4997999999999998e-05 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482829+0000", 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8820000000000003e-06 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.486550+0000", 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037209370000000001 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.486647+0000", 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6331999999999999e-05 2024-09-16T03:41:24.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.487297+0000", 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00065033799999999998 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.494616+0000", 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0073187460000000001 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.500895+0000", 2024-09-16T03:41:24.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006279438 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.500966+0000", 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1286999999999997e-05 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.4615.0:1000 3.6s0 3:64535dd7:::smithi12138290-18:head [append 0~0] snapc 26=[26,25,24,23,22,21,1f,1e,1d,1b,8] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-16T03:41:16.482841+0000", 2024-09-16T03:41:24.238 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.6384016040000002, 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.023637195, 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.4615", 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1000 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482841+0000", 2024-09-16T03:41:24.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482841+0000", 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482842+0000", 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3200000000000005e-07 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482846+0000", 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6990000000000001e-06 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482852+0000", 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.883e-06 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482858+0000", 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7769999999999999e-06 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482908+0000", 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9469999999999999e-05 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.482913+0000", 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.495570+0000", 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012656514000000001 2024-09-16T03:41:24.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.495672+0000", 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102756 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.496146+0000", 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047371199999999999 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.499031+0000", 2024-09-16T03:41:24.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028847339999999999 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.506438+0000", 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0074070459999999996 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-16T03:41:16.506478+0000", 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0608000000000002e-05 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.244 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-16T03:41:24.245 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-16T03:41:24.245 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-16T03:41:24.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:41:24.362 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:24.362 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:24.362 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:41:24.362 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:24.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:41:24.475 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:24.475 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:41:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:41:24.476 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:24.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:41:24.589 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:24.589 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:41:24.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 73/31 ECSubWrite(tid=83, reqid=osd.1.0:83, at_version=73'38, trim_to=0'0, roll_forward_to=72'36))", 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.108994662999997, 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059263110000000001, 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477863+0000", 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.17e-07 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477878+0000", 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4824000000000001e-05 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477887+0000", 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6509999999999996e-06 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477895+0000", 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.401e-06 2024-09-16T03:41:24.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477945+0000", 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0555e-05 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.477953+0000", 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.875e-06 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483761+0000", 2024-09-16T03:41:24.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058074750000000003 2024-09-16T03:41:24.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.483789+0000", 2024-09-16T03:41:24.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8313000000000001e-05 2024-09-16T03:41:24.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.594 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 73/31 ECSubWrite(tid=85, reqid=osd.1.0:85, at_version=73'40, trim_to=0'0, roll_forward_to=72'36))", 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.105697937999999, 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071150040000000003, 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481159+0000", 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481160+0000", 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:41:24.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481177+0000", 2024-09-16T03:41:24.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7368999999999999e-05 2024-09-16T03:41:24.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481186+0000", 2024-09-16T03:41:24.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1940000000000004e-06 2024-09-16T03:41:24.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481194+0000", 2024-09-16T03:41:24.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4209999999999998e-06 2024-09-16T03:41:24.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481262+0000", 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8312000000000006e-05 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481272+0000", 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3519999999999996e-06 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488236+0000", 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069643860000000004 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488274+0000", 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8390999999999999e-05 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=111, reqid=osd.6.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.095242515999999, 2024-09-16T03:41:24.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006602218, 2024-09-16T03:41:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491615+0000", 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491625+0000", 2024-09-16T03:41:24.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7249999999999992e-06 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491634+0000", 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3370000000000006e-06 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491642+0000", 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0770000000000002e-06 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491764+0000", 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012253500000000001 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491772+0000", 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1499999999999999e-06 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498186+0000", 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064136640000000003 2024-09-16T03:41:24.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498217+0000", 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1080000000000001e-05 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 73/55 ECSubWrite(tid=98, reqid=osd.10.0:98, at_version=73'24, trim_to=0'0, roll_forward_to=56'20))", 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.093376567999996, 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065373489999999996, 2024-09-16T03:41:24.605 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493481+0000", 2024-09-16T03:41:24.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.61e-07 2024-09-16T03:41:24.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493506+0000", 2024-09-16T03:41:24.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4556999999999999e-05 2024-09-16T03:41:24.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493520+0000", 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3804e-05 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493528+0000", 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9340000000000006e-06 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493606+0000", 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8144000000000003e-05 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493617+0000", 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1368e-05 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499976+0000", 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006358463 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500018+0000", 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2317999999999997e-05 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=112, reqid=osd.6.0:112, at_version=73'91, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.093052571000001, 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062537729999999998, 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.610 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493805+0000", 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5600000000000005e-07 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493823+0000", 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7804000000000001e-05 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493835+0000", 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2197e-05 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493845+0000", 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6409999999999997e-06 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493899+0000", 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4418e-05 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.493909+0000", 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5670000000000001e-06 2024-09-16T03:41:24.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500026+0000", 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006117468 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.500058+0000", 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2021999999999999e-05 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 73/31 ECSubWrite(tid=113, reqid=osd.6.0:113, at_version=73'93, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.092769683, 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083691979999999996, 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494088+0000", 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0399999999999995e-07 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494110+0000", 2024-09-16T03:41:24.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1750999999999999e-05 2024-09-16T03:41:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494122+0000", 2024-09-16T03:41:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1532000000000001e-05 2024-09-16T03:41:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494129+0000", 2024-09-16T03:41:24.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7300000000000005e-06 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495262+0000", 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011327679999999999 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.495271+0000", 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7739999999999993e-06 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502425+0000", 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071538160000000003 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502457+0000", 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2122999999999997e-05 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.619 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:682 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.059277115, 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095978390000000004, 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 682 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.620 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527580+0000", 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527582+0000", 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7910000000000001e-06 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527590+0000", 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5320000000000001e-06 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527608+0000", 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8496999999999999e-05 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527622+0000", 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3583e-05 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527699+0000", 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7799000000000005e-05 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527707+0000", 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.530874+0000", 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031660529999999998 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.530988+0000", 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114376 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.531781+0000", 2024-09-16T03:41:24.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079327 2024-09-16T03:41:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536093+0000", 2024-09-16T03:41:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043113819999999999 2024-09-16T03:41:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:24.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.537011+0000", 2024-09-16T03:41:24.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091858900000000004 2024-09-16T03:41:24.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.537178+0000", 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016688799999999999 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:683 3.es0 3:71ce6426:::smithi12138290-50:head [rollback 18] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.059202402, 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015112265, 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.629 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 683 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:24.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527655+0000", 2024-09-16T03:41:24.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:41:24.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527659+0000", 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4510000000000001e-06 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527667+0000", 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527673+0000", 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0100000000000001e-06 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527773+0000", 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100029 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527780+0000", 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3239999999999999e-06 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.532223+0000", 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044434870000000003 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.532315+0000", 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2255000000000004e-05 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533195+0000", 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087904200000000002 2024-09-16T03:41:24.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536094+0000", 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028993840000000001 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542627+0000", 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006533021 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542767+0000", 2024-09-16T03:41:24.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014024 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:684 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 19=[19,18,16,12,11,10,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.059173364999999, 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015421104, 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:24.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 684 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527684+0000", 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000005e-07 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527687+0000", 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4609999999999998e-06 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527696+0000", 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7600000000000008e-06 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527701+0000", 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3789999999999997e-06 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527803+0000", 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010234900000000001 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.527809+0000", 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2870000000000004e-06 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533474+0000", 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056649090000000001 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.533571+0000", 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7552000000000004e-05 2024-09-16T03:41:24.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.534004+0000", 2024-09-16T03:41:24.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00043247700000000001 2024-09-16T03:41:24.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.536095+0000", 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020918460000000001 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.542953+0000", 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068572249999999998 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:53.543105+0000", 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015232000000000001 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.642 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=138, reqid=osd.6.0:138, at_version=78'117, trim_to=0'0, roll_forward_to=77'113))", 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.039597607000001, 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064033809999999997, 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547260+0000", 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547261+0000", 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.032e-06 2024-09-16T03:41:24.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547287+0000", 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5905e-05 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547301+0000", 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3905e-05 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547310+0000", 2024-09-16T03:41:24.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4590000000000004e-06 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547537+0000", 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022751299999999999 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.547553+0000", 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5113000000000001e-05 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553619+0000", 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060667450000000001 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553663+0000", 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3708999999999998e-05 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s3 78/55 ECSubWrite(tid=122, reqid=osd.10.0:122, at_version=78'32, trim_to=0'0, roll_forward_to=76'30))", 2024-09-16T03:41:24.647 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.036820487, 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006109504, 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550037+0000", 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9100000000000004e-07 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550065+0000", 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7671999999999999e-05 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550075+0000", 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5540000000000003e-06 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550083+0000", 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8119999999999997e-06 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550154+0000", 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0167999999999996e-05 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550163+0000", 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7850000000000003e-06 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556099+0000", 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059356310000000002 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556146+0000", 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7391000000000001e-05 2024-09-16T03:41:24.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=139, reqid=osd.6.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.032755158000001, 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078423550000000005, 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554102+0000", 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554103+0000", 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7700000000000004e-07 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554111+0000", 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6119999999999999e-06 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554122+0000", 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0587999999999999e-05 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554129+0000", 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554193+0000", 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3522000000000006e-05 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554204+0000", 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.083e-05 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561904+0000", 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077002540000000001 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561944+0000", 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0538999999999999e-05 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.656 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.as3 78/55 ECSubWrite(tid=161, reqid=osd.12.0:161, at_version=78'43, trim_to=0'0, roll_forward_to=77'41))", 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.032034938999999, 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067129479999999998, 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554822+0000", 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554823+0000", 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554849+0000", 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6461999999999998e-05 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554862+0000", 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2918000000000001e-05 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554872+0000", 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.005e-05 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554971+0000", 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8665999999999994e-05 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554981+0000", 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0161e-05 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561484+0000", 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006503404 2024-09-16T03:41:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561535+0000", 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0934999999999999e-05 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s3 78/31 ECSubWrite(tid=133, reqid=osd.1.0:133, at_version=78'56, trim_to=0'0, roll_forward_to=76'52))", 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.031531576999999, 2024-09-16T03:41:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006360844, 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555326+0000", 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555327+0000", 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1299999999999998e-07 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555347+0000", 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0389000000000001e-05 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555359+0000", 2024-09-16T03:41:24.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2371999999999999e-05 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555368+0000", 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0480000000000004e-06 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555454+0000", 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5662999999999996e-05 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555464+0000", 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0018e-05 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561663+0000", 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061989549999999999 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.561687+0000", 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3485999999999998e-05 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.665 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.as3 78/55 ECSubWrite(tid=163, reqid=osd.12.0:163, at_version=78'45, trim_to=0'0, roll_forward_to=77'41))", 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.030651644999999, 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078281010000000005, 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556206+0000", 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556207+0000", 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0809999999999999e-06 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556231+0000", 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.406e-05 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556244+0000", 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3696999999999999e-05 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556258+0000", 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3342e-05 2024-09-16T03:41:24.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556856+0000", 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059772700000000005 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556865+0000", 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.166e-06 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564009+0000", 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007144358 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564034+0000", 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4669999999999999e-05 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s2 78/31 ECSubWrite(tid=140, reqid=osd.6.0:140, at_version=78'119, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:41:24.670 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.029242419999999, 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064168259999999996, 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557615+0000", 2024-09-16T03:41:24.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557616+0000", 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9899999999999999e-07 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557640+0000", 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4034000000000001e-05 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557651+0000", 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1382999999999999e-05 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557660+0000", 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557721+0000", 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1484999999999998e-05 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557731+0000", 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0051000000000001e-05 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.563993+0000", 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062610649999999997 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.564032+0000", 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9193999999999998e-05 2024-09-16T03:41:24.674 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:879 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.480957+0000", 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.105900349999999, 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091711669999999992, 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:24.675 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 879 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480957+0000", 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480957+0000", 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480959+0000", 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2259999999999999e-06 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480967+0000", 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.401e-06 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480983+0000", 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6704000000000001e-05 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.480993+0000", 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0013000000000001e-05 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481088+0000", 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4381000000000003e-05 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481098+0000", 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8640000000000008e-06 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.483602+0000", 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025048539999999999 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.483734+0000", 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000131617 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.484915+0000", 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001180918 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.489925+0000", 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050105779999999999 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490013+0000", 2024-09-16T03:41:24.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7682000000000005e-05 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490128+0000", 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114929 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:880 3.es0 3:71ce6426:::smithi12138290-50:head [rollback 8] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.105837523999998, 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010599224000000001, 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 880 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:24.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481020+0000", 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2799999999999995e-07 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481024+0000", 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481036+0000", 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2088e-05 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481044+0000", 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5129999999999999e-06 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481172+0000", 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012779299999999999 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481177+0000", 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.485371+0000", 2024-09-16T03:41:24.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041941410000000002 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.485473+0000", 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102734 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.486086+0000", 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061263300000000001 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490135+0000", 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040486020000000001 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491526+0000", 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001391155 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491619+0000", 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3103000000000006e-05 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:881 3.es0 3:71ce6426:::smithi12138290-50:head [append 0~0] snapc 1d=[1d,1b,1a,19,11,b,8] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.105773501000002, 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010772182999999999, 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:24.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 881 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481084+0000", 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9500000000000003e-07 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481089+0000", 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481096+0000", 2024-09-16T03:41:24.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5549999999999997e-06 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481101+0000", 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4800000000000003e-06 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481212+0000", 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111263 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:24.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.481220+0000", 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4069999999999993e-06 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.486523+0000", 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053031099999999998 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.486636+0000", 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112339 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.487220+0000", 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058448099999999998 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.490409+0000", 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031889729999999999 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491777+0000", 2024-09-16T03:41:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001367458 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:02.491856+0000", 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9231000000000001e-05 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:985 3.es0 3:7acde04a:::smithi12138290-46:head [stat,delete] snapc 25=[25,24,23,22,21,20,1f,1e,1d,1b,b,8] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.482982+0000", 2024-09-16T03:41:24.694 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.103875632999999, 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010206787, 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 985 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.482982+0000", 2024-09-16T03:41:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.482982+0000", 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.482983+0000", 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.567e-06 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.482989+0000", 2024-09-16T03:41:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1739999999999997e-06 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.483003+0000", 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3555e-05 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.483011+0000", 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1300000000000001e-06 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.483530+0000", 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051879400000000005 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.483537+0000", 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9870000000000002e-06 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.484023+0000", 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000486262 2024-09-16T03:41:24.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.484137+0000", 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011434000000000001 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.484926+0000", 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078816599999999998 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.488760+0000", 2024-09-16T03:41:24.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003834035 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.493017+0000", 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042577020000000004 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.493188+0000", 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000171075 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.700 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:24.701 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:24.701 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:24.702 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:41:24.663+0000 7f58ee745640 -1 osd.12 83 log_to_monitors true 2024-09-16T03:41:24.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:41:24.809 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:24.809 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:24.809 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:41:24.809 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:24.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:41:24.920 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:24.921 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:24.921 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:41:24.921 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:41:24.921 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:24.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:41:25.034 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 73/31 ECSubWrite(tid=57, reqid=osd.8.0:57, at_version=73'44, trim_to=0'0, roll_forward_to=63'40))", 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.480913+0000", 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.551375851000003, 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087512579999999996, 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.035 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480913+0000", 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480913+0000", 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480914+0000", 2024-09-16T03:41:25.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0699999999999996e-07 2024-09-16T03:41:25.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480938+0000", 2024-09-16T03:41:25.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3949000000000001e-05 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480949+0000", 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0984e-05 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.480957+0000", 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481866+0000", 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090959899999999998 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.481875+0000", 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9709999999999993e-06 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.489628+0000", 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077526219999999998 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.489665+0000", 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6786000000000003e-05 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.043 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=91, reqid=osd.14.0:91, at_version=73'73, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.484293+0000", 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.547995909000001, 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012749313, 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484293+0000", 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484293+0000", 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484302+0000", 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3580000000000003e-06 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484318+0000", 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6327000000000001e-05 2024-09-16T03:41:25.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484329+0000", 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0912e-05 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484336+0000", 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6320000000000008e-06 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484387+0000", 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0361000000000001e-05 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484396+0000", 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5279999999999992e-06 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497001+0000", 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012604529999999999 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497043+0000", 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1665000000000002e-05 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=92, reqid=osd.14.0:92, at_version=73'75, trim_to=0'0, roll_forward_to=72'71))", 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.547937290999997, 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01525908, 2024-09-16T03:41:25.048 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:41:25.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484352+0000", 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.22e-07 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484387+0000", 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4653999999999997e-05 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484394+0000", 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9550000000000003e-06 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.484400+0000", 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7570000000000001e-06 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488364+0000", 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039642260000000004 2024-09-16T03:41:25.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488373+0000", 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5880000000000004e-06 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499570+0000", 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011197087999999999 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499611+0000", 2024-09-16T03:41:25.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.049e-05 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 73/55 ECSubWrite(tid=177, reqid=osd.0.0:174, at_version=0'0, trim_to=0'0, roll_forward_to=73'30))", 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.488711+0000", 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.543578447000002, 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010894513, 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.053 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488711+0000", 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488711+0000", 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488712+0000", 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6099999999999999e-07 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488726+0000", 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4591e-05 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488818+0000", 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1811999999999994e-05 2024-09-16T03:41:25.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488829+0000", 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1114999999999999e-05 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488913+0000", 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3481999999999998e-05 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488922+0000", 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4069999999999999e-06 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499556+0000", 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010634403000000001 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499605+0000", 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8841999999999998e-05 2024-09-16T03:41:25.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/31 ECSubWrite(tid=111, reqid=osd.6.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.490976+0000", 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.541313303000003, 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089424069999999994, 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490976+0000", 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490976+0000", 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490977+0000", 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9199999999999999e-07 2024-09-16T03:41:25.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.490988+0000", 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1206000000000001e-05 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491004+0000", 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6351999999999999e-05 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.491013+0000", 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3489999999999992e-06 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492712+0000", 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016998219999999999 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492722+0000", 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3109999999999995e-06 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499886+0000", 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071638120000000003 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.499918+0000", 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2663000000000002e-05 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/31 ECSubWrite(tid=113, reqid=osd.6.0:113, at_version=73'93, trim_to=0'0, roll_forward_to=73'89))", 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.494706+0000", 2024-09-16T03:41:25.062 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.537582858999997, 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013014379, 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494706+0000", 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494706+0000", 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494707+0000", 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0500000000000003e-07 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494728+0000", 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1047999999999999e-05 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494742+0000", 2024-09-16T03:41:25.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3672000000000001e-05 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.494758+0000", 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6184999999999998e-05 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497457+0000", 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002698729 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497466+0000", 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.507681+0000", 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010215284999999999 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.507721+0000", 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9715000000000002e-05 2024-09-16T03:41:25.066 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 73/31 ECSubWrite(tid=95, reqid=osd.14.0:95, at_version=73'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.530225563000002, 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086042050000000002, 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502064+0000", 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0200000000000001e-07 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502084+0000", 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9769e-05 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502098+0000", 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3365e-05 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502105+0000", 2024-09-16T03:41:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7430000000000002e-06 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502860+0000", 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075463899999999998 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502869+0000", 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8619999999999992e-06 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510637+0000", 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077679530000000002 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510668+0000", 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1072000000000003e-05 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 94/55 ECSubWrite(tid=387, reqid=osd.0.0:387, at_version=94'95, trim_to=0'0, roll_forward_to=92'91))", 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.494850+0000", 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.537439607, 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0090856400000000007, 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.494850+0000", 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.494850+0000", 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.494850+0000", 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0999999999999998e-07 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.494870+0000", 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9652000000000001e-05 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.494879+0000", 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6279999999999994e-06 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.494886+0000", 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9909999999999996e-06 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.494951+0000", 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5741000000000006e-05 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.494960+0000", 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7169999999999997e-06 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.503906+0000", 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089461890000000002 2024-09-16T03:41:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.503935+0000", 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9011999999999999e-05 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 94/31 ECSubWrite(tid=232, reqid=osd.6.0:232, at_version=94'185, trim_to=0'0, roll_forward_to=92'183))", 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.495300+0000", 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.536989460000001, 2024-09-16T03:41:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089659900000000001, 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495300+0000", 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495300+0000", 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495300+0000", 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4000000000000001e-07 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495322+0000", 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1548000000000001e-05 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495332+0000", 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0027000000000001e-05 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495380+0000", 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8170000000000001e-05 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495446+0000", 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6191000000000004e-05 2024-09-16T03:41:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495455+0000", 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3680000000000002e-06 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.504235+0000", 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087798370000000004 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.504266+0000", 2024-09-16T03:41:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1109000000000003e-05 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.as2 94/85 ECSubWrite(tid=233, reqid=osd.6.0:233, at_version=94'71, trim_to=0'0, roll_forward_to=91'67))", 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.495408+0000", 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.53688079, 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088128909999999998, 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495408+0000", 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495408+0000", 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495409+0000", 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2800000000000003e-07 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495428+0000", 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9758999999999998e-05 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495436+0000", 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8609999999999998e-06 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495443+0000", 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5950000000000004e-06 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495503+0000", 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0251000000000003e-05 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495512+0000", 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.704e-06 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.504184+0000", 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086722520000000001 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.504221+0000", 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7141000000000002e-05 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 94/31 ECSubWrite(tid=214, reqid=osd.14.0:214, at_version=94'173, trim_to=0'0, roll_forward_to=94'171))", 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.495511+0000", 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.536777916, 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087121170000000001, 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495511+0000", 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495511+0000", 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495513+0000", 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495535+0000", 2024-09-16T03:41:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1488000000000002e-05 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495544+0000", 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8999999999999995e-06 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495551+0000", 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2899999999999997e-06 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495602+0000", 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1515999999999999e-05 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.495611+0000", 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6389999999999998e-06 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.504186+0000", 2024-09-16T03:41:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085749999999999993 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.504223+0000", 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7400000000000001e-05 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 94/31 ECSubWrite(tid=215, reqid=osd.14.0:215, at_version=94'175, trim_to=0'0, roll_forward_to=94'171))", 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.496271+0000", 2024-09-16T03:41:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.536018637, 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087147720000000008, 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.496271+0000", 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.496271+0000", 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.496271+0000", 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1099999999999995e-07 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.496289+0000", 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8074999999999999e-05 2024-09-16T03:41:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.496301+0000", 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1513e-05 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.496308+0000", 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8759999999999999e-06 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.499170+0000", 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028624509999999998 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.499179+0000", 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0820000000000005e-06 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.504951+0000", 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005771928 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.504985+0000", 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4236000000000002e-05 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 94/31 ECSubWrite(tid=238, reqid=osd.6.0:238, at_version=94'191, trim_to=0'0, roll_forward_to=94'187))", 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.509053+0000", 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.523236229, 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099739949999999994, 2024-09-16T03:41:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509053+0000", 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509053+0000", 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509054+0000", 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4899999999999995e-07 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509073+0000", 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.912e-05 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509095+0000", 2024-09-16T03:41:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2515999999999999e-05 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509103+0000", 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0669999999999992e-06 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509162+0000", 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8746999999999998e-05 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509173+0000", 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0898e-05 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.518991+0000", 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098178039999999994 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.519027+0000", 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6193999999999999e-05 2024-09-16T03:41:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 94/31 ECSubWrite(tid=220, reqid=osd.14.0:220, at_version=94'179, trim_to=0'0, roll_forward_to=94'175))", 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.509290+0000", 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.522999116999999, 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097411389999999994, 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509290+0000", 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509290+0000", 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509291+0000", 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8500000000000001e-07 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509316+0000", 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5130000000000002e-05 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509329+0000", 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.348e-05 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509337+0000", 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1310000000000006e-06 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509397+0000", 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9571000000000003e-05 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509406+0000", 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4789999999999996e-06 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.519005+0000", 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095991879999999998 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.519031+0000", 2024-09-16T03:41:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6475e-05 2024-09-16T03:41:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 94/31 ECSubWrite(tid=103, reqid=client.4615.0:982, at_version=94'81, trim_to=0'0, roll_forward_to=94'79))", 2024-09-16T03:41:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.519345+0000", 2024-09-16T03:41:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.512944231000001, 2024-09-16T03:41:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098081739999999994, 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.519345+0000", 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.519345+0000", 2024-09-16T03:41:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.519346+0000", 2024-09-16T03:41:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0800000000000004e-07 2024-09-16T03:41:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.524379+0000", 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050338300000000004 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.524390+0000", 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1021e-05 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.524396+0000", 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9320000000000001e-06 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.524443+0000", 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6765999999999999e-05 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.524449+0000", 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7760000000000003e-06 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.529121+0000", 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046725789999999996 2024-09-16T03:41:25.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.529153+0000", 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1662000000000003e-05 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs4 97/55 ECSubWrite(tid=411, reqid=osd.0.0:411, at_version=97'105, trim_to=0'0, roll_forward_to=97'103))", 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.499870+0000", 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.5324187029999994, 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010860913999999999, 2024-09-16T03:41:25.110 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.499870+0000", 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.499870+0000", 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.499871+0000", 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1299999999999998e-07 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.499921+0000", 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9761e-05 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.499936+0000", 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4949999999999999e-05 2024-09-16T03:41:25.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.499945+0000", 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5529999999999993e-06 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500013+0000", 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8427000000000001e-05 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500023+0000", 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0379e-05 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.510646+0000", 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010622240999999999 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.510731+0000", 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5690000000000001e-05 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.115 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s6 97/31 ECSubWrite(tid=246, reqid=osd.6.0:246, at_version=97'199, trim_to=0'0, roll_forward_to=95'195))", 2024-09-16T03:41:25.115 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.500112+0000", 2024-09-16T03:41:25.115 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.5321767210000008, 2024-09-16T03:41:25.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010668772999999999, 2024-09-16T03:41:25.115 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.115 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.115 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.115 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500112+0000", 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500112+0000", 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500113+0000", 2024-09-16T03:41:25.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4399999999999999e-07 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500133+0000", 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0123999999999999e-05 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500146+0000", 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2299e-05 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500153+0000", 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7209999999999994e-06 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500326+0000", 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000172573 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500336+0000", 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0397999999999999e-05 2024-09-16T03:41:25.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.510736+0000", 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010399406999999999 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.510781+0000", 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5507000000000003e-05 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.5s2 97/31 ECSubWrite(tid=114, reqid=osd.8.0:114, at_version=97'93, trim_to=0'0, roll_forward_to=97'91))", 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.504690+0000", 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.5275990559999997, 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.009318972, 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.504690+0000", 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.504690+0000", 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.504691+0000", 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6800000000000009e-07 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.504723+0000", 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1542999999999999e-05 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.504736+0000", 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.306e-05 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.504744+0000", 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5029999999999997e-06 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.504823+0000", 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9326000000000006e-05 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.504833+0000", 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7079999999999992e-06 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.513968+0000", 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.009134629 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.514009+0000", 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1235000000000001e-05 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 97/31 ECSubWrite(tid=241, reqid=osd.14.0:241, at_version=97'191, trim_to=0'0, roll_forward_to=97'188))", 2024-09-16T03:41:25.124 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.505093+0000", 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.5271964499999999, 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0090775449999999994, 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505093+0000", 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505093+0000", 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505094+0000", 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6099999999999999e-07 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505128+0000", 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4916000000000002e-05 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505141+0000", 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2945000000000001e-05 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505151+0000", 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3090000000000001e-06 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505217+0000", 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5847999999999997e-05 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505227+0000", 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.514133+0000", 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089058820000000004 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.514170+0000", 2024-09-16T03:41:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7166000000000003e-05 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s1 97/31 ECSubWrite(tid=242, reqid=osd.14.0:242, at_version=97'193, trim_to=0'0, roll_forward_to=97'188))", 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.506196+0000", 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.5260933219999995, 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010521927, 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.129 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.506196+0000", 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.506196+0000", 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.506197+0000", 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0800000000000003e-07 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.506220+0000", 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3570999999999998e-05 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.506233+0000", 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2702e-05 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.506241+0000", 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9300000000000003e-06 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.506289+0000", 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8210000000000001e-05 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.506295+0000", 2024-09-16T03:41:25.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8510000000000004e-06 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.516686+0000", 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010390772 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.516718+0000", 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1983000000000002e-05 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.133 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.134 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.134 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.134 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:25.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:41:25.238 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:41:25.238 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:25.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:41:25.349 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:25.349 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:25.349 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:41:25.349 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:41:25.349 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:25.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:41:25.461 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4341.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.198019802000005, 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59293511399999999, 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4341", 2024-09-16T03:41:25.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.76:0/15706134", 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260705+0000", 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260707+0000", 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.66e-06 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260713+0000", 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6150000000000002e-06 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260728+0000", 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5333e-05 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260741+0000", 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260825+0000", 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4147999999999995e-05 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.260836+0000", 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0351e-05 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852485+0000", 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59164876099999997 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852573+0000", 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8352999999999999e-05 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.852639+0000", 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853142+0000", 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050253099999999996 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853585+0000", 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000443666 2024-09-16T03:41:25.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853615+0000", 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9774e-05 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:39:58.853640+0000", 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4924999999999999e-05 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:483 3.8s0 3:1676b3ea:::smithi12138290-33:head [copy-from ver 28] snapc 13=[13,12,11,10,b,8,5] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:42.473301+0000", 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.985423814000001, 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021928366000000001, 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 483 2024-09-16T03:41:25.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473301+0000", 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473301+0000", 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473303+0000", 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1009999999999999e-06 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473315+0000", 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473363+0000", 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7976000000000003e-05 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473371+0000", 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8869999999999998e-06 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473437+0000", 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5574999999999995e-05 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473444+0000", 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6089999999999997e-06 2024-09-16T03:41:25.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.475559+0000", 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021149239999999998 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.475702+0000", 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014371699999999999 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.492468+0000", 2024-09-16T03:41:25.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016765841 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.494270+0000", 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018016639999999999 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.495153+0000", 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000883434 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.495229+0000", 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5977999999999997e-05 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:484 3.8s0 3:1676b3ea:::smithi12138290-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:42.473483+0000", 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.985241533, 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020673568, 2024-09-16T03:41:25.475 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 484 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473483+0000", 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473483+0000", 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473484+0000", 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.850000000000001e-07 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473489+0000", 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9409999999999998e-06 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473498+0000", 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9339999999999995e-06 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473505+0000", 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9070000000000001e-06 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473736+0000", 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000231179 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.473743+0000", 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.861e-06 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.476480+0000", 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027373520000000002 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.476532+0000", 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2021000000000002e-05 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.493932+0000", 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017399465999999999 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.494046+0000", 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114605 2024-09-16T03:41:25.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:42.494157+0000", 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011031699999999999 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.7s4 78/55 ECSubWrite(tid=65, reqid=osd.9.0:65, at_version=78'48, trim_to=0'0, roll_forward_to=76'46))", 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.550309+0000", 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.908415146999999, 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015216214, 2024-09-16T03:41:25.481 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550309+0000", 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550309+0000", 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550310+0000", 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550332+0000", 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1905000000000001e-05 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550341+0000", 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0759999999999992e-06 2024-09-16T03:41:25.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550350+0000", 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5010000000000004e-06 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550461+0000", 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011110199999999999 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550471+0000", 2024-09-16T03:41:25.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0351e-05 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565488+0000", 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015016715999999999 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565526+0000", 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8003999999999997e-05 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs2 78/31 ECSubWrite(tid=123, reqid=osd.10.0:123, at_version=78'60, trim_to=0'0, roll_forward_to=76'54))", 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.550551+0000", 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.908173206000001, 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014556872, 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550551+0000", 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550551+0000", 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550552+0000", 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.01e-07 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550576+0000", 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4239e-05 2024-09-16T03:41:25.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550585+0000", 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8780000000000004e-06 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550593+0000", 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8499999999999994e-06 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550809+0000", 2024-09-16T03:41:25.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000216154 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.550820+0000", 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1032e-05 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565062+0000", 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014242322 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565108+0000", 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5995999999999998e-05 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs3 78/55 ECSubWrite(tid=232, reqid=osd.0.0:232, at_version=78'45, trim_to=0'0, roll_forward_to=76'43))", 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.551319+0000", 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.907405343000001, 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014559028999999999, 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.490 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551319+0000", 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551319+0000", 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551320+0000", 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551348+0000", 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8458000000000001e-05 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551812+0000", 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046333000000000002 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551823+0000", 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551934+0000", 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111041 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.551943+0000", 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2159999999999995e-06 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565839+0000", 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013895573 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.565878+0000", 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9416999999999999e-05 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.494 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.fs3 78/55 ECSubWrite(tid=233, reqid=osd.0.0:233, at_version=78'47, trim_to=0'0, roll_forward_to=76'43))", 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.552032+0000", 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.906692823, 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016044706999999998, 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552032+0000", 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552032+0000", 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552033+0000", 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.78e-07 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552057+0000", 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3810000000000001e-05 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552069+0000", 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2051e-05 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552078+0000", 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3559999999999999e-06 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552594+0000", 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051604000000000005 2024-09-16T03:41:25.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552603+0000", 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1880000000000008e-06 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568033+0000", 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015429481 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568077+0000", 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3803000000000001e-05 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s4 78/31 ECSubWrite(tid=119, reqid=osd.14.0:119, at_version=78'103, trim_to=0'0, roll_forward_to=77'101))", 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.552465+0000", 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.906259901999999, 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015710608000000001, 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.499 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552465+0000", 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552465+0000", 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552465+0000", 2024-09-16T03:41:25.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1899999999999998e-07 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552493+0000", 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7671e-05 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552503+0000", 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7610000000000002e-06 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552511+0000", 2024-09-16T03:41:25.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8920000000000006e-06 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552597+0000", 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5118999999999996e-05 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.552605+0000", 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9109999999999993e-06 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568142+0000", 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015537268 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568175+0000", 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3566999999999999e-05 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s5 78/31 ECSubWrite(tid=131, reqid=osd.1.0:131, at_version=78'54, trim_to=0'0, roll_forward_to=76'52))", 2024-09-16T03:41:25.503 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.553878+0000", 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.904847126, 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014243021999999999, 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553878+0000", 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553878+0000", 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553878+0000", 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4699999999999998e-07 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553938+0000", 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0383000000000001e-05 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553954+0000", 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5807999999999998e-05 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553964+0000", 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6020000000000006e-06 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554032+0000", 2024-09-16T03:41:25.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8257999999999996e-05 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554039+0000", 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4699999999999996e-06 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568066+0000", 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014026733 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568121+0000", 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4521000000000002e-05 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.9s1 78/55 ECSubWrite(tid=132, reqid=osd.1.0:132, at_version=78'64, trim_to=0'0, roll_forward_to=77'62))", 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.553990+0000", 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.904734998999999, 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014899674, 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.508 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553990+0000", 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553990+0000", 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.553990+0000", 2024-09-16T03:41:25.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2900000000000004e-07 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554010+0000", 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9592000000000001e-05 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554018+0000", 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.636e-06 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554026+0000", 2024-09-16T03:41:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4440000000000002e-06 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554096+0000", 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0170000000000001e-05 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554103+0000", 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9179999999999997e-06 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568859+0000", 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014756132999999999 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568889+0000", 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0252000000000001e-05 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s3 78/31 ECSubWrite(tid=139, reqid=osd.6.0:138, at_version=0'0, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.554150+0000", 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.904574880999998, 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014696384, 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554150+0000", 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554150+0000", 2024-09-16T03:41:25.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554151+0000", 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554162+0000", 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1537e-05 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554177+0000", 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4235e-05 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554185+0000", 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1790000000000008e-06 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554433+0000", 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024847500000000001 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.554444+0000", 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.039e-05 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568773+0000", 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014329328000000001 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.568846+0000", 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3178999999999998e-05 2024-09-16T03:41:25.516 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.0s5 78/31 ECSubWrite(tid=133, reqid=osd.1.0:133, at_version=78'56, trim_to=0'0, roll_forward_to=76'52))", 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555550+0000", 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.903174275000001, 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015172118, 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555550+0000", 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555550+0000", 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555551+0000", 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555575+0000", 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3643999999999999e-05 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555601+0000", 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6574e-05 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555613+0000", 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1225e-05 2024-09-16T03:41:25.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566114+0000", 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010501310999999999 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566128+0000", 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4099e-05 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570627+0000", 2024-09-16T03:41:25.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044985989999999998 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570722+0000", 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5829999999999996e-05 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.9s1 78/55 ECSubWrite(tid=134, reqid=osd.1.0:134, at_version=78'66, trim_to=0'0, roll_forward_to=77'62))", 2024-09-16T03:41:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555670+0000", 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.903054257000001, 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015094101, 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555670+0000", 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555670+0000", 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555671+0000", 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3599999999999999e-07 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555733+0000", 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1939000000000004e-05 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555745+0000", 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2125e-05 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555761+0000", 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6169000000000001e-05 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566189+0000", 2024-09-16T03:41:25.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010428181 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566198+0000", 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4849999999999992e-06 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570680+0000", 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044819810000000003 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570764+0000", 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4784999999999995e-05 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.es1 78/55 ECSubWrite(tid=38, reqid=osd.3.0:38, at_version=78'57, trim_to=0'0, roll_forward_to=76'53))", 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.555707+0000", 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.903017478999999, 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014962289, 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.526 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555707+0000", 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555707+0000", 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555708+0000", 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0270000000000001e-06 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555735+0000", 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.667e-05 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555747+0000", 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2546e-05 2024-09-16T03:41:25.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.555767+0000", 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9230000000000001e-05 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566201+0000", 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010434173999999999 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566211+0000", 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0064999999999999e-05 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570621+0000", 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044097099999999998 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570669+0000", 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8866999999999998e-05 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.530 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s4 78/31 ECSubWrite(tid=120, reqid=osd.14.0:120, at_version=78'105, trim_to=0'0, roll_forward_to=77'101))", 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.556218+0000", 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.902506705, 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014634856, 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556218+0000", 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556218+0000", 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556219+0000", 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0100000000000001e-06 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556243+0000", 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4111999999999999e-05 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556259+0000", 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5766000000000001e-05 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.556268+0000", 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3100000000000006e-06 2024-09-16T03:41:25.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566183+0000", 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099146909999999998 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566193+0000", 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0249e-05 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570674+0000", 2024-09-16T03:41:25.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044805599999999998 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570853+0000", 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017915800000000001 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.1s3 78/31 ECSubWrite(tid=140, reqid=osd.6.0:140, at_version=78'119, trim_to=0'0, roll_forward_to=78'117))", 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:57.557327+0000", 2024-09-16T03:41:25.535 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.901397594999999, 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013347019, 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557327+0000", 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557327+0000", 2024-09-16T03:41:25.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557328+0000", 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4799999999999997e-07 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557352+0000", 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4025000000000001e-05 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557368+0000", 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6402999999999999e-05 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.557378+0000", 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9869999999999994e-06 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566126+0000", 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087474890000000007 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.566136+0000", 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0261000000000001e-05 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570628+0000", 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044920439999999997 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:57.570674+0000", 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5961999999999998e-05 2024-09-16T03:41:25.539 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:1002 3.2s0 3:43c65f68:::smithi12138290-36:head [read 0~1884160,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:17.482017+0000", 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.9767079890000003, 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013493771, 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:41:25.540 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1002 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482017+0000", 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482017+0000", 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482019+0000", 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3580000000000001e-06 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482029+0000", 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9650000000000003e-06 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482044+0000", 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5495e-05 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482053+0000", 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6640000000000004e-06 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482127+0000", 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3454999999999994e-05 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482135+0000", 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3499999999999997e-06 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.485095+0000", 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029597949999999999 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.485211+0000", 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115899 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.495510+0000", 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01029979 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:1003 3.2s0 3:43c65f68:::smithi12138290-36:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:41:25.545 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:17.482176+0000", 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.9765484989999997, 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014541662, 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1003 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482176+0000", 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482176+0000", 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482177+0000", 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9700000000000005e-07 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482183+0000", 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6180000000000003e-06 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482193+0000", 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0096e-05 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482199+0000", 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7039999999999997e-06 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482380+0000", 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000180972 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482388+0000", 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4479999999999997e-06 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.485670+0000", 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003282264 2024-09-16T03:41:25.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.485785+0000", 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114673 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.496718+0000", 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01093299 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:1004 3.2s0 3:43c65f68:::smithi12138290-36:head [read 0~1884160,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:17.482379+0000", 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.9763456860000002, 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016136859, 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1004 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.551 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482379+0000", 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482379+0000", 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482380+0000", 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482386+0000", 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8279999999999999e-06 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482394+0000", 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8949999999999998e-06 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482400+0000", 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5440000000000004e-06 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482507+0000", 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107373 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.482514+0000", 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7050000000000002e-06 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.486314+0000", 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00379963 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.486408+0000", 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4720000000000001e-05 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.498516+0000", 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012107332 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.555 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.9s1 98/55 ECSubWrite(tid=229, reqid=osd.1.0:229, at_version=98'106, trim_to=0'0, roll_forward_to=97'102))", 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:17.485118+0000", 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.9736063069999998, 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012965964999999999, 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.485118+0000", 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.485118+0000", 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.485119+0000", 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0699999999999997e-07 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.485147+0000", 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8575e-05 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.485156+0000", 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9069999999999996e-06 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.485164+0000", 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8229999999999995e-06 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.488252+0000", 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003088188 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.488262+0000", 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.498059+0000", 2024-09-16T03:41:25.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097968320000000001 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:17.498084+0000", 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5592000000000002e-05 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.560 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:25.560 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:41:25.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:41:25.566 INFO:tasks.ceph.osd.12.smithi076.stderr:2024-09-16T03:41:25.564+0000 7f58e24e3640 -1 osd.12 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:41:25.666 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:25.666 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:25.666 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-16T03:41:25.666 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:25.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:41:25.778 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:25.778 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-16T03:41:25.778 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-16T03:41:25.778 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-16T03:41:25.778 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:25.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:41:25.892 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-16T03:41:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-16T03:41:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-16T03:41:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-16T03:41:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:375 3.3s0 3:c98c73f3:::smithi12138290-40:head [append 0~671744] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:41:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.352104351000001, 2024-09-16T03:41:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086953299999999994, 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 375 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:41:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537290+0000", 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537291+0000", 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.59e-06 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537708+0000", 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041628699999999999 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537721+0000", 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3738e-05 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537728+0000", 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4409999999999998e-06 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537798+0000", 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0543000000000005e-05 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.537931+0000", 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013217700000000001 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.539173+0000", 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012421660000000001 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.540584+0000", 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001411673 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545889+0000", 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053041340000000003 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545985+0000", 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6581000000000002e-05 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:376 3.3s0 3:c98c73f3:::smithi12138290-40:head [append 0~770048] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.347388090000003, 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087274550000000003, 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 376 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542006+0000", 2024-09-16T03:41:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542008+0000", 2024-09-16T03:41:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.756e-06 2024-09-16T03:41:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542428+0000", 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041977300000000002 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542440+0000", 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2894000000000001e-05 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542447+0000", 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.584e-06 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542503+0000", 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5483999999999999e-05 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542630+0000", 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012754500000000001 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.543864+0000", 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012343090000000001 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545743+0000", 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018785799999999999 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550638+0000", 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048951790000000004 2024-09-16T03:41:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550733+0000", 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5351000000000002e-05 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:377 3.3s0 3:c98c73f3:::smithi12138290-40:head [setxattr _header (57) in=7b] snapc d=[d,b,8,5,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.346932021999997, 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084428420000000007, 2024-09-16T03:41:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 377 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542462+0000", 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542463+0000", 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2600000000000002e-07 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542467+0000", 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6040000000000003e-06 2024-09-16T03:41:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542473+0000", 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6019999999999999e-06 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542476+0000", 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.393e-06 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.544778+0000", 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023019329999999999 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.544876+0000", 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7448000000000006e-05 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.545337+0000", 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046103399999999998 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.547435+0000", 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020983320000000001 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550831+0000", 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033956920000000001 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.550905+0000", 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4078000000000005e-05 2024-09-16T03:41:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4615.0:378 3.3s0 3:c98c73f3:::smithi12138290-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.346895600000003, 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011059941, 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4615", 2024-09-16T03:41:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.121:0/3747589546", 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 378 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542499+0000", 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1699999999999998e-07 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542502+0000", 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8040000000000002e-06 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542509+0000", 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3320000000000003e-06 2024-09-16T03:41:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.542515+0000", 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.49e-06 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.546046+0000", 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035316670000000001 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.546144+0000", 2024-09-16T03:41:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8040000000000003e-05 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.551270+0000", 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005125597 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.551384+0000", 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011363599999999999 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:32.553558+0000", 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021747580000000002 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=95, reqid=client.4615.0:649, at_version=73'44, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.400901304999998, 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097259600000000005, 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488493+0000", 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488494+0000", 2024-09-16T03:41:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5099999999999998e-07 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488724+0000", 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000229764 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488736+0000", 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.221e-05 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.488744+0000", 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3110000000000006e-06 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492660+0000", 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039159759999999998 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.492669+0000", 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3449999999999995e-06 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498200+0000", 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055302169999999996 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.498219+0000", 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9185999999999999e-05 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=100, reqid=client.4615.0:652, at_version=73'47, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.392321768999999, 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083283569999999998, 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497072+0000", 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497073+0000", 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.61e-07 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497223+0000", 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014987799999999999 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497236+0000", 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3278999999999999e-05 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497245+0000", 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8130000000000002e-06 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497329+0000", 2024-09-16T03:41:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3671999999999994e-05 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.497338+0000", 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7000000000000003e-06 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.505360+0000", 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080216100000000002 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.505401+0000", 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0744000000000002e-05 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 73/31 ECSubWrite(tid=102, reqid=osd.10.0:102, at_version=73'50, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.387463119000003, 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087707529999999992, 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501931+0000", 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501932+0000", 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3e-07 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501949+0000", 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7711e-05 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501962+0000", 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2549e-05 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.501969+0000", 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1720000000000001e-06 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502019+0000", 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0340000000000003e-05 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502028+0000", 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8570000000000001e-06 2024-09-16T03:41:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510672+0000", 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086435399999999999 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510702+0000", 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9954000000000002e-05 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 73/31 ECSubWrite(tid=95, reqid=osd.14.0:95, at_version=73'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.387308349000001, 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085979139999999999, 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502086+0000", 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3099999999999998e-07 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502103+0000", 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6769000000000002e-05 2024-09-16T03:41:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502114+0000", 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1258e-05 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502121+0000", 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6429999999999998e-06 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502175+0000", 2024-09-16T03:41:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3698e-05 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.502183+0000", 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6589999999999996e-06 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510650+0000", 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084665680000000007 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:40:52.510684+0000", 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3887999999999998e-05 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 94/31 ECSubWrite(tid=220, reqid=osd.14.0:220, at_version=94'179, trim_to=0'0, roll_forward_to=94'175))", 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.509341+0000", 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.380052932, 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097268170000000004, 2024-09-16T03:41:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509341+0000", 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509341+0000", 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509342+0000", 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6400000000000001e-07 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509372+0000", 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9620000000000001e-05 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509381+0000", 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6220000000000004e-06 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509388+0000", 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.888e-06 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509458+0000", 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0407999999999994e-05 2024-09-16T03:41:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509467+0000", 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2840000000000006e-06 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.519039+0000", 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095719480000000003 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.519068+0000", 2024-09-16T03:41:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9283000000000001e-05 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.cs2 94/88 ECSubWrite(tid=393, reqid=osd.0.0:393, at_version=94'199, trim_to=0'0, roll_forward_to=94'195))", 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:13.509381+0000", 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.380012898, 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097088769999999994, 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509381+0000", 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509381+0000", 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509382+0000", 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:41:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509402+0000", 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9936e-05 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509414+0000", 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2605e-05 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509422+0000", 2024-09-16T03:41:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7389999999999999e-06 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509476+0000", 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3940000000000002e-05 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.509485+0000", 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6300000000000004e-06 2024-09-16T03:41:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.519065+0000", 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095800699999999996 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:13.519090+0000", 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5233e-05 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.es6 97/55 ECSubWrite(tid=67, reqid=osd.3.0:67, at_version=97'87, trim_to=0'0, roll_forward_to=95'85))", 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.486484+0000", 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.4029097630000003, 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014108595, 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486484+0000", 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486484+0000", 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486485+0000", 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.92e-07 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486507+0000", 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2938999999999998e-05 2024-09-16T03:41:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486518+0000", 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0181e-05 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486528+0000", 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0027000000000001e-05 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486619+0000", 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1337999999999999e-05 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486629+0000", 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0069e-05 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500559+0000", 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013929736 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500593+0000", 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4112999999999997e-05 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.es6 97/55 ECSubWrite(tid=68, reqid=osd.3.0:68, at_version=97'89, trim_to=0'0, roll_forward_to=95'85))", 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.486665+0000", 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.4027289320000005, 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013977107000000001, 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486665+0000", 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486665+0000", 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486666+0000", 2024-09-16T03:41:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2600000000000002e-07 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486704+0000", 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8015000000000001e-05 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486714+0000", 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0353000000000001e-05 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.486723+0000", 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4130000000000006e-06 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.487881+0000", 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011579159999999999 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.487892+0000", 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0977e-05 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500624+0000", 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012731991999999999 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500642+0000", 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8715e-05 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 97/31 ECSubWrite(tid=234, reqid=client.4615.0:999, at_version=97'184, trim_to=0'0, roll_forward_to=95'181))", 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.487991+0000", 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.4014029509999997, 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01262772, 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.487991+0000", 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.487991+0000", 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.487992+0000", 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3799999999999996e-07 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.488079+0000", 2024-09-16T03:41:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6688999999999996e-05 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.488092+0000", 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2700999999999999e-05 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.488101+0000", 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.448e-06 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.488391+0000", 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028985000000000003 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.488401+0000", 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0379e-05 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500598+0000", 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012196479 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500619+0000", 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1236000000000001e-05 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.cs2 97/88 ECSubWrite(tid=407, reqid=osd.0.0:407, at_version=97'211, trim_to=0'0, roll_forward_to=95'209))", 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.489755+0000", 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3996395100000001, 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01177623, 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.489755+0000", 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.489755+0000", 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.489755+0000", 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1100000000000002e-07 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.489781+0000", 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5609000000000002e-05 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.489794+0000", 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3730999999999999e-05 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.489806+0000", 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2174999999999999e-05 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.489889+0000", 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2503999999999995e-05 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.489906+0000", 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7428999999999999e-05 2024-09-16T03:41:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.501489+0000", 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011583082999999999 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.501531+0000", 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1387999999999997e-05 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.cs2 97/88 ECSubWrite(tid=408, reqid=osd.0.0:408, at_version=97'213, trim_to=0'0, roll_forward_to=95'209))", 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.490242+0000", 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3991522399999994, 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012073245999999999, 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.490242+0000", 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.490242+0000", 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.490243+0000", 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5499999999999996e-07 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.490268+0000", 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4890999999999999e-05 2024-09-16T03:41:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.490282+0000", 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3974e-05 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.490291+0000", 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2730000000000008e-06 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.497572+0000", 2024-09-16T03:41:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072812659999999998 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.497583+0000", 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1044e-05 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.502281+0000", 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046976789999999997 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.502315+0000", 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4164e-05 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.4s4 97/85 ECSubWrite(tid=166, reqid=osd.10.0:166, at_version=97'44, trim_to=0'0, roll_forward_to=94'42))", 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.492508+0000", 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3968863129999995, 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012844482000000001, 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492508+0000", 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492508+0000", 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492508+0000", 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7800000000000001e-07 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492535+0000", 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7114000000000002e-05 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492548+0000", 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2707999999999999e-05 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492558+0000", 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9389999999999992e-06 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500838+0000", 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082801990000000002 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500847+0000", 2024-09-16T03:41:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2830000000000007e-06 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505320+0000", 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044724359999999998 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505352+0000", 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2625e-05 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.bs3 97/31 ECSubWrite(tid=167, reqid=osd.10.0:167, at_version=97'80, trim_to=0'0, roll_forward_to=95'78))", 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.492574+0000", 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3968197870000001, 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0090381430000000002, 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492574+0000", 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492574+0000", 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492575+0000", 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6899999999999998e-07 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492591+0000", 2024-09-16T03:41:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6705e-05 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492600+0000", 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6209999999999993e-06 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.492607+0000", 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.493285+0000", 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00067794100000000005 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.493296+0000", 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0556e-05 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.501590+0000", 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082942229999999999 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.501612+0000", 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2748e-05 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 97/31 ECSubWrite(tid=236, reqid=osd.14.0:236, at_version=97'188, trim_to=0'0, roll_forward_to=95'181))", 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.493457+0000", 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3959370260000004, 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088371950000000008, 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.493457+0000", 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.493457+0000", 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.493458+0000", 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5099999999999998e-07 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.493481+0000", 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3489000000000002e-05 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.493493+0000", 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759999999999999e-05 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.493506+0000", 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2492e-05 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.497495+0000", 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039895699999999996 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.497505+0000", 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4429999999999992e-06 2024-09-16T03:41:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.502252+0000", 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047475699999999996 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.502294+0000", 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2020000000000001e-05 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 97/31 ECSubWrite(tid=239, reqid=client.4615.0:1000, at_version=97'189, trim_to=0'0, roll_forward_to=97'183))", 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.496175+0000", 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3932193739999992, 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089014290000000006, 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.496175+0000", 2024-09-16T03:41:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.496175+0000", 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.496176+0000", 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.057e-06 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.496198+0000", 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2439999999999999e-05 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.496218+0000", 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9304999999999999e-05 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.496227+0000", 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7450000000000007e-06 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500685+0000", 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044581129999999997 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.500695+0000", 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5429999999999999e-06 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505035+0000", 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043398559999999996 2024-09-16T03:41:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.505076+0000", 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1369999999999999e-05 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDECSubOpWrite(3.6s2 97/31 ECSubWrite(tid=245, reqid=osd.14.0:245, at_version=97'197, trim_to=0'0, roll_forward_to=97'193))", 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-16T03:41:16.519406+0000", 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3699879480000003, 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094869289999999998, 2024-09-16T03:41:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.519406+0000", 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.519406+0000", 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.519407+0000", 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8299999999999997e-07 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.519420+0000", 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3281999999999999e-05 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.519430+0000", 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7559999999999994e-06 2024-09-16T03:41:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.519435+0000", 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1220000000000003e-06 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.521120+0000", 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016849320000000001 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_started", 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.521127+0000", 2024-09-16T03:41:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9469999999999997e-06 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_committed", 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.528857+0000", 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077299659999999996 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-16T03:41:16.528893+0000", 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6440999999999999e-05 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-16T03:41:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-16T03:41:25.988 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-16T03:41:25.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:41:26.104 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:26.104 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:26.104 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-09-16T03:41:26.104 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:26.113 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:41:26.216 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:26.217 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-09-16T03:41:26.217 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-09-16T03:41:26.217 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-09-16T03:41:26.217 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:26.225 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:41:26.329 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T03:41:26.330 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-09-16T03:41:26.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-09-16T03:41:26.330 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-09-16T03:41:26.330 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T03:41:26.552 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T03:41:26.570 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:41:27.561 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T03:41:28.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:41:28.755 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:41:28.756 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T03:41:28.756 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T03:41:28.756 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:41:28.764 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-09-16T03:41:33.765 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T03:41:33.765 INFO:tasks.thrashosds.thrasher:Adding osd 12 2024-09-16T03:41:33.766 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 12 2024-09-16T03:41:34.070 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T03:41:34.068+0000 7f011120c640 -1 mon.a@0(leader).osd e108 definitely_dead 0 2024-09-16T03:41:34.578 INFO:teuthology.orchestra.run.smithi076.stderr:marked in osd.12. 2024-09-16T03:41:34.592 INFO:tasks.thrashosds.thrasher:Added osd 12 2024-09-16T03:41:34.592 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T03:41:34.592 INFO:tasks.thrashosds.thrasher:Adding osd 13 2024-09-16T03:41:34.593 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 13 2024-09-16T03:41:34.877 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T03:41:34.876+0000 7f011120c640 -1 mon.a@0(leader).osd e109 definitely_dead 0 2024-09-16T03:41:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:marked in osd.13. 2024-09-16T03:41:35.599 INFO:tasks.thrashosds.thrasher:Added osd 13 2024-09-16T03:41:35.599 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T03:41:35.719 INFO:teuthology.orchestra.run.smithi076.stdout:mon.a: {} 2024-09-16T03:41:35.719 INFO:teuthology.orchestra.run.smithi076.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:41:35.728 INFO:teuthology.orchestra.run.smithi076.stdout:mon.b: {} 2024-09-16T03:41:35.728 INFO:teuthology.orchestra.run.smithi076.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:41:35.737 INFO:teuthology.orchestra.run.smithi076.stdout:mon.c: {} 2024-09-16T03:41:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:41:35.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T03:41:36.589 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T03:41:36.606 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-16T03:41:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T03:41:37.608 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T03:41:38.599 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T03:41:38.615 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-16T03:41:39.610 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T03:41:39.624 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T03:41:40.611 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T03:41:40.628 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-16T03:41:41.620 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-16T03:41:41.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T03:41:42.622 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T03:41:42.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T03:41:43.635 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T03:41:43.649 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T03:41:44.637 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T03:41:44.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-16T03:41:45.647 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T03:41:45.663 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T03:41:46.657 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T03:41:46.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-16T03:41:47.665 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-16T03:41:47.682 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T03:41:48.674 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T03:41:48.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T03:41:49.680 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T03:41:49.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T03:41:50.685 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T03:41:50.702 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-16T03:41:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T03:41:51.706 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T03:41:52.697 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T03:41:52.712 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-16T03:41:53.703 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-16T03:41:53.719 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T03:41:54.712 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T03:41:54.726 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-16T03:41:55.720 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-16T03:41:55.734 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T03:41:56.721 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T03:41:56.737 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-16T03:41:57.729 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T03:41:57.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T03:41:58.734 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T03:41:58.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T03:41:59.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T03:42:00.745 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T03:42:00.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-16T03:42:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-16T03:42:01.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T03:42:02.765 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T03:42:02.783 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-16T03:42:03.770 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-16T03:42:03.787 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-16T03:42:04.776 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.13 to 1 (10000) 2024-09-16T03:42:04.793 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-16T03:42:05.781 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-16T03:42:05.799 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T03:42:06.787 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T03:42:06.803 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-16T03:42:07.794 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-16T03:42:07.809 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T03:42:07.809 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T03:42:07.809 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T03:42:07.809 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T03:42:07.809 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T03:42:07.809 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:42:08.087 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:42:08.088 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":142,"fsid":"ecf7a42a-e941-4b24-8693-0ff8b6eb41b1","created":"2024-09-16T03:39:42.686123+0000","modified":"2024-09-16T03:42:07.792264+0000","last_up_change":"2024-09-16T03:41:26.550004+0000","last_in_change":"2024-09-16T03:41:35.577075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:39:56.952065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:39:57.213209+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":27,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0100002288818359,"score_stable":4.0100002288818359,"optimal_score":0.81000000238418579,"raw_score_acting":3.25,"raw_score_stable":3.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8820e1f9-b619-44a5-abb1-7cfa97fcef00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6811","nonce":2834882614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6813","nonce":2834882614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6817","nonce":2834882614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6815","nonce":2834882614}]},"public_addr":"172.21.15.76:6811/2834882614","cluster_addr":"172.21.15.76:6813/2834882614","heartbeat_back_addr":"172.21.15.76:6817/2834882614","heartbeat_front_addr":"172.21.15.76:6815/2834882614","state":["exists","up"]},{"osd":1,"uuid":"0f58ae45-e2ff-4c4f-9875-7d34ec312cbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6809","nonce":2045143946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6811","nonce":2045143946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6815","nonce":2045143946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6813","nonce":2045143946}]},"public_addr":"172.21.15.103:6809/2045143946","cluster_addr":"172.21.15.103:6811/2045143946","heartbeat_back_addr":"172.21.15.103:6815/2045143946","heartbeat_front_addr":"172.21.15.103:6813/2045143946","state":["exists","up"]},{"osd":2,"uuid":"b369fe92-86b8-45b3-a418-024f0e4becd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6817","nonce":273257286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6819","nonce":273257286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6823","nonce":273257286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6821","nonce":273257286}]},"public_addr":"172.21.15.119:6817/273257286","cluster_addr":"172.21.15.119:6819/273257286","heartbeat_back_addr":"172.21.15.119:6823/273257286","heartbeat_front_addr":"172.21.15.119:6821/273257286","state":["exists","up"]},{"osd":3,"uuid":"b9521bf1-46ef-402c-8472-c1b476280b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6811","nonce":278480145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6815","nonce":278480145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6823","nonce":278480145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6819","nonce":278480145}]},"public_addr":"172.21.15.121:6811/278480145","cluster_addr":"172.21.15.121:6815/278480145","heartbeat_back_addr":"172.21.15.121:6823/278480145","heartbeat_front_addr":"172.21.15.121:6819/278480145","state":["exists","up"]},{"osd":4,"uuid":"15bec6a1-fc13-4822-8cfa-899b83886c10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6803","nonce":3302785479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6805","nonce":3302785479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6809","nonce":3302785479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6807","nonce":3302785479}]},"public_addr":"172.21.15.76:6803/3302785479","cluster_addr":"172.21.15.76:6805/3302785479","heartbeat_back_addr":"172.21.15.76:6809/3302785479","heartbeat_front_addr":"172.21.15.76:6807/3302785479","state":["exists","up"]},{"osd":5,"uuid":"e9b234ec-bfa4-4f72-adab-295e067b2e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6801","nonce":3765230361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6803","nonce":3765230361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6807","nonce":3765230361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6805","nonce":3765230361}]},"public_addr":"172.21.15.103:6801/3765230361","cluster_addr":"172.21.15.103:6803/3765230361","heartbeat_back_addr":"172.21.15.103:6807/3765230361","heartbeat_front_addr":"172.21.15.103:6805/3765230361","state":["exists","up"]},{"osd":6,"uuid":"6d4ae9fb-8d8f-451d-a12f-1265302b03ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6825","nonce":446165121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6827","nonce":446165121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6831","nonce":446165121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6829","nonce":446165121}]},"public_addr":"172.21.15.119:6825/446165121","cluster_addr":"172.21.15.119:6827/446165121","heartbeat_back_addr":"172.21.15.119:6831/446165121","heartbeat_front_addr":"172.21.15.119:6829/446165121","state":["exists","up"]},{"osd":7,"uuid":"2c9456ed-00ec-46aa-814e-f85eb441fc4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6810","nonce":1304791608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6814","nonce":1304791608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6822","nonce":1304791608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6818","nonce":1304791608}]},"public_addr":"172.21.15.121:6810/1304791608","cluster_addr":"172.21.15.121:6814/1304791608","heartbeat_back_addr":"172.21.15.121:6822/1304791608","heartbeat_front_addr":"172.21.15.121:6818/1304791608","state":["exists","up"]},{"osd":8,"uuid":"b618a02a-8628-48f5-bb5f-f7591aafa118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6819","nonce":2630853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6821","nonce":2630853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6825","nonce":2630853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6823","nonce":2630853906}]},"public_addr":"172.21.15.76:6819/2630853906","cluster_addr":"172.21.15.76:6821/2630853906","heartbeat_back_addr":"172.21.15.76:6825/2630853906","heartbeat_front_addr":"172.21.15.76:6823/2630853906","state":["exists","up"]},{"osd":9,"uuid":"dc3e5be7-f012-47e6-ba35-0e3354ca0c18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6817","nonce":2702548675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6819","nonce":2702548675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6823","nonce":2702548675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6821","nonce":2702548675}]},"public_addr":"172.21.15.103:6817/2702548675","cluster_addr":"172.21.15.103:6819/2702548675","heartbeat_back_addr":"172.21.15.103:6823/2702548675","heartbeat_front_addr":"172.21.15.103:6821/2702548675","state":["exists","up"]},{"osd":10,"uuid":"4e70a7a8-7ba8-42ab-a45a-e0564b561619","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6809","nonce":205142394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6811","nonce":205142394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6815","nonce":205142394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6813","nonce":205142394}]},"public_addr":"172.21.15.119:6809/205142394","cluster_addr":"172.21.15.119:6811/205142394","heartbeat_back_addr":"172.21.15.119:6815/205142394","heartbeat_front_addr":"172.21.15.119:6813/205142394","state":["exists","up"]},{"osd":11,"uuid":"d16bc493-335f-41c2-bd68-cbad93c49a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6825","nonce":1551091359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6827","nonce":1551091359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6831","nonce":1551091359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6829","nonce":1551091359}]},"public_addr":"172.21.15.121:6825/1551091359","cluster_addr":"172.21.15.121:6827/1551091359","heartbeat_back_addr":"172.21.15.121:6831/1551091359","heartbeat_front_addr":"172.21.15.121:6829/1551091359","state":["exists","up"]},{"osd":12,"uuid":"a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":83,"up_from":107,"up_thru":55,"down_at":84,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6827","nonce":217312017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6829","nonce":217312017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6833","nonce":217312017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6831","nonce":217312017}]},"public_addr":"172.21.15.76:6827/217312017","cluster_addr":"172.21.15.76:6829/217312017","heartbeat_back_addr":"172.21.15.76:6833/217312017","heartbeat_front_addr":"172.21.15.76:6831/217312017","state":["exists","up"]},{"osd":13,"uuid":"61e9204b-8269-4c35-b409-5cfbd5e6abbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":53,"up_from":76,"up_thru":110,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6825","nonce":3364003535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6827","nonce":3364003535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6831","nonce":3364003535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6829","nonce":3364003535}]},"public_addr":"172.21.15.103:6825/3364003535","cluster_addr":"172.21.15.103:6827/3364003535","heartbeat_back_addr":"172.21.15.103:6831/3364003535","heartbeat_front_addr":"172.21.15.103:6829/3364003535","state":["exists","up"]},{"osd":14,"uuid":"0bd5338e-eb99-44cc-9f5f-1c8ef903d02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6801","nonce":1147733023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6803","nonce":1147733023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6807","nonce":1147733023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6805","nonce":1147733023}]},"public_addr":"172.21.15.119:6801/1147733023","cluster_addr":"172.21.15.119:6803/1147733023","heartbeat_back_addr":"172.21.15.119:6807/1147733023","heartbeat_front_addr":"172.21.15.119:6805/1147733023","state":["exists","up"]},{"osd":15,"uuid":"f240405e-d268-41cc-bd99-5a2ca71a85a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6801","nonce":3149764080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6803","nonce":3149764080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6807","nonce":3149764080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6805","nonce":3149764080}]},"public_addr":"172.21.15.121:6801/3149764080","cluster_addr":"172.21.15.121:6803/3149764080","heartbeat_back_addr":"172.21.15.121:6807/3149764080","heartbeat_front_addr":"172.21.15.121:6805/3149764080","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.895331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:12.029403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:21.537323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:59.887564+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:01.297243+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:00.164464+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.217756+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:58.131838+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:03.346738+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:11.711895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:19.361985+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:24.073827+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:25.680752+0000","dead_epoch":84},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:11.065521+0000","dead_epoch":54},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:09.217323+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.187965+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.2","osds":[14,3]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":11,"length":1},{"begin":33,"length":1},{"begin":38,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:42:08.100 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T03:42:08.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:42:08.380 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:42:08.381 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":142,"fsid":"ecf7a42a-e941-4b24-8693-0ff8b6eb41b1","created":"2024-09-16T03:39:42.686123+0000","modified":"2024-09-16T03:42:07.792264+0000","last_up_change":"2024-09-16T03:41:26.550004+0000","last_in_change":"2024-09-16T03:41:35.577075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:39:56.952065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:39:57.213209+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":27,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0100002288818359,"score_stable":4.0100002288818359,"optimal_score":0.81000000238418579,"raw_score_acting":3.25,"raw_score_stable":3.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8820e1f9-b619-44a5-abb1-7cfa97fcef00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6811","nonce":2834882614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6813","nonce":2834882614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6817","nonce":2834882614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6815","nonce":2834882614}]},"public_addr":"172.21.15.76:6811/2834882614","cluster_addr":"172.21.15.76:6813/2834882614","heartbeat_back_addr":"172.21.15.76:6817/2834882614","heartbeat_front_addr":"172.21.15.76:6815/2834882614","state":["exists","up"]},{"osd":1,"uuid":"0f58ae45-e2ff-4c4f-9875-7d34ec312cbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6809","nonce":2045143946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6811","nonce":2045143946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6815","nonce":2045143946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6813","nonce":2045143946}]},"public_addr":"172.21.15.103:6809/2045143946","cluster_addr":"172.21.15.103:6811/2045143946","heartbeat_back_addr":"172.21.15.103:6815/2045143946","heartbeat_front_addr":"172.21.15.103:6813/2045143946","state":["exists","up"]},{"osd":2,"uuid":"b369fe92-86b8-45b3-a418-024f0e4becd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6817","nonce":273257286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6819","nonce":273257286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6823","nonce":273257286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6821","nonce":273257286}]},"public_addr":"172.21.15.119:6817/273257286","cluster_addr":"172.21.15.119:6819/273257286","heartbeat_back_addr":"172.21.15.119:6823/273257286","heartbeat_front_addr":"172.21.15.119:6821/273257286","state":["exists","up"]},{"osd":3,"uuid":"b9521bf1-46ef-402c-8472-c1b476280b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6811","nonce":278480145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6815","nonce":278480145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6823","nonce":278480145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6819","nonce":278480145}]},"public_addr":"172.21.15.121:6811/278480145","cluster_addr":"172.21.15.121:6815/278480145","heartbeat_back_addr":"172.21.15.121:6823/278480145","heartbeat_front_addr":"172.21.15.121:6819/278480145","state":["exists","up"]},{"osd":4,"uuid":"15bec6a1-fc13-4822-8cfa-899b83886c10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6803","nonce":3302785479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6805","nonce":3302785479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6809","nonce":3302785479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6807","nonce":3302785479}]},"public_addr":"172.21.15.76:6803/3302785479","cluster_addr":"172.21.15.76:6805/3302785479","heartbeat_back_addr":"172.21.15.76:6809/3302785479","heartbeat_front_addr":"172.21.15.76:6807/3302785479","state":["exists","up"]},{"osd":5,"uuid":"e9b234ec-bfa4-4f72-adab-295e067b2e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6801","nonce":3765230361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6803","nonce":3765230361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6807","nonce":3765230361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6805","nonce":3765230361}]},"public_addr":"172.21.15.103:6801/3765230361","cluster_addr":"172.21.15.103:6803/3765230361","heartbeat_back_addr":"172.21.15.103:6807/3765230361","heartbeat_front_addr":"172.21.15.103:6805/3765230361","state":["exists","up"]},{"osd":6,"uuid":"6d4ae9fb-8d8f-451d-a12f-1265302b03ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6825","nonce":446165121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6827","nonce":446165121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6831","nonce":446165121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6829","nonce":446165121}]},"public_addr":"172.21.15.119:6825/446165121","cluster_addr":"172.21.15.119:6827/446165121","heartbeat_back_addr":"172.21.15.119:6831/446165121","heartbeat_front_addr":"172.21.15.119:6829/446165121","state":["exists","up"]},{"osd":7,"uuid":"2c9456ed-00ec-46aa-814e-f85eb441fc4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6810","nonce":1304791608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6814","nonce":1304791608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6822","nonce":1304791608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6818","nonce":1304791608}]},"public_addr":"172.21.15.121:6810/1304791608","cluster_addr":"172.21.15.121:6814/1304791608","heartbeat_back_addr":"172.21.15.121:6822/1304791608","heartbeat_front_addr":"172.21.15.121:6818/1304791608","state":["exists","up"]},{"osd":8,"uuid":"b618a02a-8628-48f5-bb5f-f7591aafa118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6819","nonce":2630853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6821","nonce":2630853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6825","nonce":2630853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6823","nonce":2630853906}]},"public_addr":"172.21.15.76:6819/2630853906","cluster_addr":"172.21.15.76:6821/2630853906","heartbeat_back_addr":"172.21.15.76:6825/2630853906","heartbeat_front_addr":"172.21.15.76:6823/2630853906","state":["exists","up"]},{"osd":9,"uuid":"dc3e5be7-f012-47e6-ba35-0e3354ca0c18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6817","nonce":2702548675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6819","nonce":2702548675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6823","nonce":2702548675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6821","nonce":2702548675}]},"public_addr":"172.21.15.103:6817/2702548675","cluster_addr":"172.21.15.103:6819/2702548675","heartbeat_back_addr":"172.21.15.103:6823/2702548675","heartbeat_front_addr":"172.21.15.103:6821/2702548675","state":["exists","up"]},{"osd":10,"uuid":"4e70a7a8-7ba8-42ab-a45a-e0564b561619","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6809","nonce":205142394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6811","nonce":205142394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6815","nonce":205142394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6813","nonce":205142394}]},"public_addr":"172.21.15.119:6809/205142394","cluster_addr":"172.21.15.119:6811/205142394","heartbeat_back_addr":"172.21.15.119:6815/205142394","heartbeat_front_addr":"172.21.15.119:6813/205142394","state":["exists","up"]},{"osd":11,"uuid":"d16bc493-335f-41c2-bd68-cbad93c49a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6825","nonce":1551091359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6827","nonce":1551091359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6831","nonce":1551091359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6829","nonce":1551091359}]},"public_addr":"172.21.15.121:6825/1551091359","cluster_addr":"172.21.15.121:6827/1551091359","heartbeat_back_addr":"172.21.15.121:6831/1551091359","heartbeat_front_addr":"172.21.15.121:6829/1551091359","state":["exists","up"]},{"osd":12,"uuid":"a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":83,"up_from":107,"up_thru":55,"down_at":84,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6827","nonce":217312017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6829","nonce":217312017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6833","nonce":217312017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6831","nonce":217312017}]},"public_addr":"172.21.15.76:6827/217312017","cluster_addr":"172.21.15.76:6829/217312017","heartbeat_back_addr":"172.21.15.76:6833/217312017","heartbeat_front_addr":"172.21.15.76:6831/217312017","state":["exists","up"]},{"osd":13,"uuid":"61e9204b-8269-4c35-b409-5cfbd5e6abbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":53,"up_from":76,"up_thru":110,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6825","nonce":3364003535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6827","nonce":3364003535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6831","nonce":3364003535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6829","nonce":3364003535}]},"public_addr":"172.21.15.103:6825/3364003535","cluster_addr":"172.21.15.103:6827/3364003535","heartbeat_back_addr":"172.21.15.103:6831/3364003535","heartbeat_front_addr":"172.21.15.103:6829/3364003535","state":["exists","up"]},{"osd":14,"uuid":"0bd5338e-eb99-44cc-9f5f-1c8ef903d02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6801","nonce":1147733023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6803","nonce":1147733023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6807","nonce":1147733023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6805","nonce":1147733023}]},"public_addr":"172.21.15.119:6801/1147733023","cluster_addr":"172.21.15.119:6803/1147733023","heartbeat_back_addr":"172.21.15.119:6807/1147733023","heartbeat_front_addr":"172.21.15.119:6805/1147733023","state":["exists","up"]},{"osd":15,"uuid":"f240405e-d268-41cc-bd99-5a2ca71a85a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6801","nonce":3149764080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6803","nonce":3149764080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6807","nonce":3149764080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6805","nonce":3149764080}]},"public_addr":"172.21.15.121:6801/3149764080","cluster_addr":"172.21.15.121:6803/3149764080","heartbeat_back_addr":"172.21.15.121:6807/3149764080","heartbeat_front_addr":"172.21.15.121:6805/3149764080","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.895331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:12.029403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:21.537323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:59.887564+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:01.297243+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:00.164464+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.217756+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:58.131838+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:03.346738+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:11.711895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:19.361985+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:24.073827+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:25.680752+0000","dead_epoch":84},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:11.065521+0000","dead_epoch":54},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:09.217323+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.187965+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.2","osds":[14,3]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":11,"length":1},{"begin":33,"length":1},{"begin":38,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:42:08.394 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T03:42:08.394 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T03:42:08.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T03:42:08.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T03:42:08.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T03:42:08.396 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T03:42:08.396 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T03:42:08.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T03:42:08.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T03:42:08.398 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T03:42:08.398 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T03:42:08.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T03:42:08.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-16T03:42:08.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-16T03:42:08.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-16T03:42:08.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-16T03:42:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:08.793 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:42:08.811 INFO:teuthology.orchestra.run.smithi076.stdout:98784247838 2024-09-16T03:42:08.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:42:08.829 INFO:teuthology.orchestra.run.smithi076.stdout:94489280542 2024-09-16T03:42:08.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:42:08.847 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:08.847 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:42:08.852 INFO:teuthology.orchestra.run.smithi076.stdout:98784247838 2024-09-16T03:42:08.853 INFO:teuthology.orchestra.run.smithi076.stdout:326417514513 2024-09-16T03:42:08.854 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T03:42:08.855 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T03:42:08.867 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:08.868 INFO:teuthology.orchestra.run.smithi076.stdout:94489280541 2024-09-16T03:42:08.869 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T03:42:08.869 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T03:42:08.882 INFO:teuthology.orchestra.run.smithi076.stdout:90194313246 2024-09-16T03:42:08.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T03:42:08.886 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:08.886 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T03:42:08.887 INFO:teuthology.orchestra.run.smithi076.stdout:459561500683 2024-09-16T03:42:08.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T03:42:08.890 INFO:teuthology.orchestra.run.smithi076.stdout:98784247838 2024-09-16T03:42:08.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T03:42:08.908 INFO:teuthology.orchestra.run.smithi076.stdout:94489280542 2024-09-16T03:42:08.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T03:42:08.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T03:42:08.918+0000 7fcbedce4640 0 --2- 172.21.15.76:0/425783450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcbd4004800 0x7fcbd4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T03:42:08.941 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:08.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T03:42:09.030 INFO:teuthology.orchestra.run.smithi076.stdout:98784247837 2024-09-16T03:42:09.030 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:42:09.121 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:09.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:42:09.665 INFO:teuthology.orchestra.run.smithi076.stdout:98784247836 2024-09-16T03:42:09.709 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247838 got 98784247836 for osd.2 2024-09-16T03:42:09.906 INFO:teuthology.orchestra.run.smithi076.stdout:459561500683 2024-09-16T03:42:09.909 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:09.938 INFO:tasks.ceph.ceph_manager.ceph:need seq 459561500683 got 459561500683 for osd.12 2024-09-16T03:42:09.938 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:09.939 INFO:teuthology.orchestra.run.smithi076.stdout:94489280542 2024-09-16T03:42:09.945 INFO:teuthology.orchestra.run.smithi076.stdout:94489280542 2024-09-16T03:42:09.956 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247839 got 98784247839 for osd.7 2024-09-16T03:42:09.957 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:09.972 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280542 for osd.6 2024-09-16T03:42:09.972 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:09.986 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:09.988 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:09.994 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280542 for osd.1 2024-09-16T03:42:09.995 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:10.013 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247839 got 98784247839 for osd.3 2024-09-16T03:42:10.014 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:10.022 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247839 got 98784247839 for osd.4 2024-09-16T03:42:10.022 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:10.026 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:10.029 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:10.029 INFO:teuthology.orchestra.run.smithi076.stdout:94489280541 2024-09-16T03:42:10.032 INFO:teuthology.orchestra.run.smithi076.stdout:326417514513 2024-09-16T03:42:10.055 INFO:tasks.ceph.ceph_manager.ceph:need seq 326417514513 got 326417514513 for osd.13 2024-09-16T03:42:10.055 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.10 2024-09-16T03:42:10.055 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:10.055 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:10.058 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247839 got 98784247839 for osd.15 2024-09-16T03:42:10.058 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:10.060 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247839 got 98784247839 for osd.8 2024-09-16T03:42:10.060 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:10.068 INFO:teuthology.orchestra.run.smithi076.stdout:98784247838 2024-09-16T03:42:10.074 INFO:teuthology.orchestra.run.smithi076.stdout:90194313246 2024-09-16T03:42:10.082 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247838 got 98784247838 for osd.9 2024-09-16T03:42:10.082 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:10.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313246 for osd.14 2024-09-16T03:42:10.092 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:10.099 INFO:teuthology.orchestra.run.smithi076.stdout:98784247838 2024-09-16T03:42:10.109 INFO:teuthology.orchestra.run.smithi076.stdout:98784247838 2024-09-16T03:42:10.113 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247838 got 98784247838 for osd.11 2024-09-16T03:42:10.113 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:10.117 INFO:teuthology.orchestra.run.smithi076.stdout:98784247836 2024-09-16T03:42:10.123 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247839 got 98784247838 for osd.0 2024-09-16T03:42:10.131 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247837 got 98784247836 for osd.5 2024-09-16T03:42:10.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:42:10.995 INFO:teuthology.orchestra.run.smithi076.stdout:98784247838 2024-09-16T03:42:11.007 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247838 got 98784247838 for osd.2 2024-09-16T03:42:11.008 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:11.123 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:42:11.131 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:42:11.414 INFO:teuthology.orchestra.run.smithi076.stdout:98784247838 2024-09-16T03:42:11.431 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247839 got 98784247838 for osd.0 2024-09-16T03:42:11.462 INFO:teuthology.orchestra.run.smithi076.stdout:98784247836 2024-09-16T03:42:11.475 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247837 got 98784247836 for osd.5 2024-09-16T03:42:12.432 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:42:12.477 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:42:12.740 INFO:teuthology.orchestra.run.smithi076.stdout:98784247839 2024-09-16T03:42:12.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247839 got 98784247839 for osd.0 2024-09-16T03:42:12.754 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:12.775 INFO:teuthology.orchestra.run.smithi076.stdout:98784247837 2024-09-16T03:42:12.788 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247837 got 98784247837 for osd.5 2024-09-16T03:42:12.789 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:12.789 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T03:42:12.789 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:13.107 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:42:13.107 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:42:13.119 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T03:42:12.940554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":539504,"kb_used_data":34840,"kb_used_omap":29,"kb_used_meta":504290,"kb_avail":1509409936,"statfs":{"total":1546188226560,"available":1545635774464,"internally_reserved":0,"allocated":35676160,"data_stored":21432310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30242,"internal_metadata":516393438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.153433"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":446,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.311209+0000","last_change":"2024-09-16T03:41:29.486608+0000","last_active":"2024-09-16T03:42:09.311209+0000","last_peered":"2024-09-16T03:42:09.311209+0000","last_clean":"2024-09-16T03:42:09.311209+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:42:09.311209+0000","last_undegraded":"2024-09-16T03:42:09.311209+0000","last_fullsized":"2024-09-16T03:42:09.311209+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T03:41:29.486447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:29.486447+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:35.026906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":256,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.311207+0000","last_change":"2024-09-16T03:41:30.473366+0000","last_active":"2024-09-16T03:42:09.311207+0000","last_peered":"2024-09-16T03:42:09.311207+0000","last_clean":"2024-09-16T03:42:09.311207+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:42:09.311207+0000","last_undegraded":"2024-09-16T03:42:09.311207+0000","last_fullsized":"2024-09-16T03:42:09.311207+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'1","last_scrub_stamp":"2024-09-16T03:41:30.473251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:30.473251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:54.704969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":252,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175564+0000","last_change":"2024-09-16T03:41:27.442141+0000","last_active":"2024-09-16T03:42:09.175564+0000","last_peered":"2024-09-16T03:42:09.175564+0000","last_clean":"2024-09-16T03:42:09.175564+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:42:09.175564+0000","last_undegraded":"2024-09-16T03:42:09.175564+0000","last_fullsized":"2024-09-16T03:42:09.175564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.442041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.442041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:42.891354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":253,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175139+0000","last_change":"2024-09-16T03:41:27.654794+0000","last_active":"2024-09-16T03:42:09.175139+0000","last_peered":"2024-09-16T03:42:09.175139+0000","last_clean":"2024-09-16T03:42:09.175139+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:42:09.175139+0000","last_undegraded":"2024-09-16T03:42:09.175139+0000","last_fullsized":"2024-09-16T03:42:09.175139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.654635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.654635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:43.580094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"100'3","reported_seq":215,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175068+0000","last_change":"2024-09-16T03:41:20.484846+0000","last_active":"2024-09-16T03:42:09.175068+0000","last_peered":"2024-09-16T03:42:09.175068+0000","last_clean":"2024-09-16T03:42:09.175068+0000","last_became_active":"2024-09-16T03:40:24.325471+0000","last_became_peered":"2024-09-16T03:40:24.325471+0000","last_unstale":"2024-09-16T03:42:09.175068+0000","last_undegraded":"2024-09-16T03:42:09.175068+0000","last_fullsized":"2024-09-16T03:42:09.175068+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'3","last_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_deep_scrub":"100'3","last_deep_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_clean_scrub_stamp":"2024-09-16T03:41:20.484718+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.412321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":254,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:08.792177+0000","last_change":"2024-09-16T03:41:19.857960+0000","last_active":"2024-09-16T03:42:08.792177+0000","last_peered":"2024-09-16T03:42:08.792177+0000","last_clean":"2024-09-16T03:42:08.792177+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:42:08.792177+0000","last_undegraded":"2024-09-16T03:42:08.792177+0000","last_fullsized":"2024-09-16T03:42:08.792177+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:19.857848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:19.857848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:23.649882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":263,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:10.040719+0000","last_change":"2024-09-16T03:42:10.040769+0000","last_active":"2024-09-16T03:42:10.040719+0000","last_peered":"2024-09-16T03:42:10.040719+0000","last_clean":"2024-09-16T03:42:10.040719+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:42:10.040719+0000","last_undegraded":"2024-09-16T03:42:10.040719+0000","last_fullsized":"2024-09-16T03:42:10.040719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:10.040579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:42:10.040579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175076+0000","last_change":"2024-09-16T03:41:55.910523+0000","last_active":"2024-09-16T03:42:09.175076+0000","last_peered":"2024-09-16T03:42:09.175076+0000","last_clean":"2024-09-16T03:42:09.175076+0000","last_became_active":"2024-09-16T03:41:55.910222+0000","last_became_peered":"2024-09-16T03:41:55.910222+0000","last_unstale":"2024-09-16T03:42:09.175076+0000","last_undegraded":"2024-09-16T03:42:09.175076+0000","last_fullsized":"2024-09-16T03:42:09.175076+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_clean_scrub_stamp":"2024-09-16T03:41:07.602473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:28.092968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":252,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175675+0000","last_change":"2024-09-16T03:41:16.470343+0000","last_active":"2024-09-16T03:42:09.175675+0000","last_peered":"2024-09-16T03:42:09.175675+0000","last_clean":"2024-09-16T03:42:09.175675+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:42:09.175675+0000","last_undegraded":"2024-09-16T03:42:09.175675+0000","last_fullsized":"2024-09-16T03:42:09.175675+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:16.470256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:16.470256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:29.234901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,5,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":76,"seq":326417514514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29268,"kb_used_data":2112,"kb_used_omap":3,"kb_used_meta":27132,"kb_avail":94342572,"statfs":{"total":96636764160,"available":96606793728,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3529,"internal_metadata":27783735},"hb_peers":[0,3,4,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":107,"seq":459561500684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29080,"kb_used_data":2112,"kb_used_omap":4,"kb_used_meta":26939,"kb_avail":94342760,"statfs":{"total":96636764160,"available":96606986240,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4385,"internal_metadata":27586271},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29144,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342696,"statfs":{"total":96636764160,"available":96606920704,"internally_reserved":0,"allocated":2166784,"data_stored":1273420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33308,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[3,5,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2166784,"data_stored":1273420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247840,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33824,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338016,"statfs":{"total":96636764160,"available":96602128384,"internally_reserved":0,"allocated":2695168,"data_stored":1802329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,5,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247840,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,5,6,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51872,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319968,"statfs":{"total":96636764160,"available":96583647232,"internally_reserved":0,"allocated":2695168,"data_stored":1802329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:13.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:42:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:42:13.409 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T03:42:12.940554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":539504,"kb_used_data":34840,"kb_used_omap":29,"kb_used_meta":504290,"kb_avail":1509409936,"statfs":{"total":1546188226560,"available":1545635774464,"internally_reserved":0,"allocated":35676160,"data_stored":21432310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30242,"internal_metadata":516393438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.153433"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":446,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.311209+0000","last_change":"2024-09-16T03:41:29.486608+0000","last_active":"2024-09-16T03:42:09.311209+0000","last_peered":"2024-09-16T03:42:09.311209+0000","last_clean":"2024-09-16T03:42:09.311209+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:42:09.311209+0000","last_undegraded":"2024-09-16T03:42:09.311209+0000","last_fullsized":"2024-09-16T03:42:09.311209+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T03:41:29.486447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:29.486447+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:35.026906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":256,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.311207+0000","last_change":"2024-09-16T03:41:30.473366+0000","last_active":"2024-09-16T03:42:09.311207+0000","last_peered":"2024-09-16T03:42:09.311207+0000","last_clean":"2024-09-16T03:42:09.311207+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:42:09.311207+0000","last_undegraded":"2024-09-16T03:42:09.311207+0000","last_fullsized":"2024-09-16T03:42:09.311207+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'1","last_scrub_stamp":"2024-09-16T03:41:30.473251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:30.473251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:54.704969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":252,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175564+0000","last_change":"2024-09-16T03:41:27.442141+0000","last_active":"2024-09-16T03:42:09.175564+0000","last_peered":"2024-09-16T03:42:09.175564+0000","last_clean":"2024-09-16T03:42:09.175564+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:42:09.175564+0000","last_undegraded":"2024-09-16T03:42:09.175564+0000","last_fullsized":"2024-09-16T03:42:09.175564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.442041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.442041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:42.891354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":253,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175139+0000","last_change":"2024-09-16T03:41:27.654794+0000","last_active":"2024-09-16T03:42:09.175139+0000","last_peered":"2024-09-16T03:42:09.175139+0000","last_clean":"2024-09-16T03:42:09.175139+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:42:09.175139+0000","last_undegraded":"2024-09-16T03:42:09.175139+0000","last_fullsized":"2024-09-16T03:42:09.175139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.654635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.654635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:43.580094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"100'3","reported_seq":215,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175068+0000","last_change":"2024-09-16T03:41:20.484846+0000","last_active":"2024-09-16T03:42:09.175068+0000","last_peered":"2024-09-16T03:42:09.175068+0000","last_clean":"2024-09-16T03:42:09.175068+0000","last_became_active":"2024-09-16T03:40:24.325471+0000","last_became_peered":"2024-09-16T03:40:24.325471+0000","last_unstale":"2024-09-16T03:42:09.175068+0000","last_undegraded":"2024-09-16T03:42:09.175068+0000","last_fullsized":"2024-09-16T03:42:09.175068+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'3","last_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_deep_scrub":"100'3","last_deep_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_clean_scrub_stamp":"2024-09-16T03:41:20.484718+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.412321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":254,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:08.792177+0000","last_change":"2024-09-16T03:41:19.857960+0000","last_active":"2024-09-16T03:42:08.792177+0000","last_peered":"2024-09-16T03:42:08.792177+0000","last_clean":"2024-09-16T03:42:08.792177+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:42:08.792177+0000","last_undegraded":"2024-09-16T03:42:08.792177+0000","last_fullsized":"2024-09-16T03:42:08.792177+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:19.857848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:19.857848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:23.649882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":263,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:10.040719+0000","last_change":"2024-09-16T03:42:10.040769+0000","last_active":"2024-09-16T03:42:10.040719+0000","last_peered":"2024-09-16T03:42:10.040719+0000","last_clean":"2024-09-16T03:42:10.040719+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:42:10.040719+0000","last_undegraded":"2024-09-16T03:42:10.040719+0000","last_fullsized":"2024-09-16T03:42:10.040719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:10.040579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:42:10.040579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175076+0000","last_change":"2024-09-16T03:41:55.910523+0000","last_active":"2024-09-16T03:42:09.175076+0000","last_peered":"2024-09-16T03:42:09.175076+0000","last_clean":"2024-09-16T03:42:09.175076+0000","last_became_active":"2024-09-16T03:41:55.910222+0000","last_became_peered":"2024-09-16T03:41:55.910222+0000","last_unstale":"2024-09-16T03:42:09.175076+0000","last_undegraded":"2024-09-16T03:42:09.175076+0000","last_fullsized":"2024-09-16T03:42:09.175076+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_clean_scrub_stamp":"2024-09-16T03:41:07.602473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:28.092968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":252,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175675+0000","last_change":"2024-09-16T03:41:16.470343+0000","last_active":"2024-09-16T03:42:09.175675+0000","last_peered":"2024-09-16T03:42:09.175675+0000","last_clean":"2024-09-16T03:42:09.175675+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:42:09.175675+0000","last_undegraded":"2024-09-16T03:42:09.175675+0000","last_fullsized":"2024-09-16T03:42:09.175675+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:16.470256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:16.470256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:29.234901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,5,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":76,"seq":326417514514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29268,"kb_used_data":2112,"kb_used_omap":3,"kb_used_meta":27132,"kb_avail":94342572,"statfs":{"total":96636764160,"available":96606793728,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3529,"internal_metadata":27783735},"hb_peers":[0,3,4,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":107,"seq":459561500684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29080,"kb_used_data":2112,"kb_used_omap":4,"kb_used_meta":26939,"kb_avail":94342760,"statfs":{"total":96636764160,"available":96606986240,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4385,"internal_metadata":27586271},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29144,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342696,"statfs":{"total":96636764160,"available":96606920704,"internally_reserved":0,"allocated":2166784,"data_stored":1273420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33308,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[3,5,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2166784,"data_stored":1273420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247840,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33824,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338016,"statfs":{"total":96636764160,"available":96602128384,"internally_reserved":0,"allocated":2695168,"data_stored":1802329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,5,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247840,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,5,6,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51872,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319968,"statfs":{"total":96636764160,"available":96583647232,"internally_reserved":0,"allocated":2695168,"data_stored":1802329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:13.410 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T03:42:13.410 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-16T03:42:13.709 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T03:42:13.718 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T03:42:13.719 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:13.995 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:42:13.995 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:42:14.008 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T03:42:12.940554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":539504,"kb_used_data":34840,"kb_used_omap":29,"kb_used_meta":504290,"kb_avail":1509409936,"statfs":{"total":1546188226560,"available":1545635774464,"internally_reserved":0,"allocated":35676160,"data_stored":21432310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30242,"internal_metadata":516393438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.153433"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":446,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.311209+0000","last_change":"2024-09-16T03:41:29.486608+0000","last_active":"2024-09-16T03:42:09.311209+0000","last_peered":"2024-09-16T03:42:09.311209+0000","last_clean":"2024-09-16T03:42:09.311209+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:42:09.311209+0000","last_undegraded":"2024-09-16T03:42:09.311209+0000","last_fullsized":"2024-09-16T03:42:09.311209+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T03:41:29.486447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:29.486447+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:35.026906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":256,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.311207+0000","last_change":"2024-09-16T03:41:30.473366+0000","last_active":"2024-09-16T03:42:09.311207+0000","last_peered":"2024-09-16T03:42:09.311207+0000","last_clean":"2024-09-16T03:42:09.311207+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:42:09.311207+0000","last_undegraded":"2024-09-16T03:42:09.311207+0000","last_fullsized":"2024-09-16T03:42:09.311207+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'1","last_scrub_stamp":"2024-09-16T03:41:30.473251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:30.473251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:54.704969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":252,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175564+0000","last_change":"2024-09-16T03:41:27.442141+0000","last_active":"2024-09-16T03:42:09.175564+0000","last_peered":"2024-09-16T03:42:09.175564+0000","last_clean":"2024-09-16T03:42:09.175564+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:42:09.175564+0000","last_undegraded":"2024-09-16T03:42:09.175564+0000","last_fullsized":"2024-09-16T03:42:09.175564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.442041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.442041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:42.891354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":253,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175139+0000","last_change":"2024-09-16T03:41:27.654794+0000","last_active":"2024-09-16T03:42:09.175139+0000","last_peered":"2024-09-16T03:42:09.175139+0000","last_clean":"2024-09-16T03:42:09.175139+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:42:09.175139+0000","last_undegraded":"2024-09-16T03:42:09.175139+0000","last_fullsized":"2024-09-16T03:42:09.175139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.654635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.654635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:43.580094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"100'3","reported_seq":215,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175068+0000","last_change":"2024-09-16T03:41:20.484846+0000","last_active":"2024-09-16T03:42:09.175068+0000","last_peered":"2024-09-16T03:42:09.175068+0000","last_clean":"2024-09-16T03:42:09.175068+0000","last_became_active":"2024-09-16T03:40:24.325471+0000","last_became_peered":"2024-09-16T03:40:24.325471+0000","last_unstale":"2024-09-16T03:42:09.175068+0000","last_undegraded":"2024-09-16T03:42:09.175068+0000","last_fullsized":"2024-09-16T03:42:09.175068+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'3","last_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_deep_scrub":"100'3","last_deep_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_clean_scrub_stamp":"2024-09-16T03:41:20.484718+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.412321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":254,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:08.792177+0000","last_change":"2024-09-16T03:41:19.857960+0000","last_active":"2024-09-16T03:42:08.792177+0000","last_peered":"2024-09-16T03:42:08.792177+0000","last_clean":"2024-09-16T03:42:08.792177+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:42:08.792177+0000","last_undegraded":"2024-09-16T03:42:08.792177+0000","last_fullsized":"2024-09-16T03:42:08.792177+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:19.857848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:19.857848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:23.649882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":263,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:10.040719+0000","last_change":"2024-09-16T03:42:10.040769+0000","last_active":"2024-09-16T03:42:10.040719+0000","last_peered":"2024-09-16T03:42:10.040719+0000","last_clean":"2024-09-16T03:42:10.040719+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:42:10.040719+0000","last_undegraded":"2024-09-16T03:42:10.040719+0000","last_fullsized":"2024-09-16T03:42:10.040719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:10.040579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:42:10.040579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175076+0000","last_change":"2024-09-16T03:41:55.910523+0000","last_active":"2024-09-16T03:42:09.175076+0000","last_peered":"2024-09-16T03:42:09.175076+0000","last_clean":"2024-09-16T03:42:09.175076+0000","last_became_active":"2024-09-16T03:41:55.910222+0000","last_became_peered":"2024-09-16T03:41:55.910222+0000","last_unstale":"2024-09-16T03:42:09.175076+0000","last_undegraded":"2024-09-16T03:42:09.175076+0000","last_fullsized":"2024-09-16T03:42:09.175076+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_clean_scrub_stamp":"2024-09-16T03:41:07.602473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:28.092968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":252,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175675+0000","last_change":"2024-09-16T03:41:16.470343+0000","last_active":"2024-09-16T03:42:09.175675+0000","last_peered":"2024-09-16T03:42:09.175675+0000","last_clean":"2024-09-16T03:42:09.175675+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:42:09.175675+0000","last_undegraded":"2024-09-16T03:42:09.175675+0000","last_fullsized":"2024-09-16T03:42:09.175675+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:16.470256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:16.470256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:29.234901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,5,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":76,"seq":326417514514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29268,"kb_used_data":2112,"kb_used_omap":3,"kb_used_meta":27132,"kb_avail":94342572,"statfs":{"total":96636764160,"available":96606793728,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3529,"internal_metadata":27783735},"hb_peers":[0,3,4,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":107,"seq":459561500684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29080,"kb_used_data":2112,"kb_used_omap":4,"kb_used_meta":26939,"kb_avail":94342760,"statfs":{"total":96636764160,"available":96606986240,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4385,"internal_metadata":27586271},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29144,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342696,"statfs":{"total":96636764160,"available":96606920704,"internally_reserved":0,"allocated":2166784,"data_stored":1273420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33308,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[3,5,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2166784,"data_stored":1273420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247840,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33824,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338016,"statfs":{"total":96636764160,"available":96602128384,"internally_reserved":0,"allocated":2695168,"data_stored":1802329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,5,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247840,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,5,6,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51872,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319968,"statfs":{"total":96636764160,"available":96583647232,"internally_reserved":0,"allocated":2695168,"data_stored":1802329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:14.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:14.286 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:42:14.286 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:42:14.298 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T03:42:12.940554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":539504,"kb_used_data":34840,"kb_used_omap":29,"kb_used_meta":504290,"kb_avail":1509409936,"statfs":{"total":1546188226560,"available":1545635774464,"internally_reserved":0,"allocated":35676160,"data_stored":21432310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30242,"internal_metadata":516393438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.153433"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":446,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.311209+0000","last_change":"2024-09-16T03:41:29.486608+0000","last_active":"2024-09-16T03:42:09.311209+0000","last_peered":"2024-09-16T03:42:09.311209+0000","last_clean":"2024-09-16T03:42:09.311209+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:42:09.311209+0000","last_undegraded":"2024-09-16T03:42:09.311209+0000","last_fullsized":"2024-09-16T03:42:09.311209+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T03:41:29.486447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:29.486447+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:35.026906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":256,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.311207+0000","last_change":"2024-09-16T03:41:30.473366+0000","last_active":"2024-09-16T03:42:09.311207+0000","last_peered":"2024-09-16T03:42:09.311207+0000","last_clean":"2024-09-16T03:42:09.311207+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:42:09.311207+0000","last_undegraded":"2024-09-16T03:42:09.311207+0000","last_fullsized":"2024-09-16T03:42:09.311207+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'1","last_scrub_stamp":"2024-09-16T03:41:30.473251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:30.473251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:54.704969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":252,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175564+0000","last_change":"2024-09-16T03:41:27.442141+0000","last_active":"2024-09-16T03:42:09.175564+0000","last_peered":"2024-09-16T03:42:09.175564+0000","last_clean":"2024-09-16T03:42:09.175564+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:42:09.175564+0000","last_undegraded":"2024-09-16T03:42:09.175564+0000","last_fullsized":"2024-09-16T03:42:09.175564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.442041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.442041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:42.891354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":253,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175139+0000","last_change":"2024-09-16T03:41:27.654794+0000","last_active":"2024-09-16T03:42:09.175139+0000","last_peered":"2024-09-16T03:42:09.175139+0000","last_clean":"2024-09-16T03:42:09.175139+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:42:09.175139+0000","last_undegraded":"2024-09-16T03:42:09.175139+0000","last_fullsized":"2024-09-16T03:42:09.175139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.654635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.654635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:43.580094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"100'3","reported_seq":215,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175068+0000","last_change":"2024-09-16T03:41:20.484846+0000","last_active":"2024-09-16T03:42:09.175068+0000","last_peered":"2024-09-16T03:42:09.175068+0000","last_clean":"2024-09-16T03:42:09.175068+0000","last_became_active":"2024-09-16T03:40:24.325471+0000","last_became_peered":"2024-09-16T03:40:24.325471+0000","last_unstale":"2024-09-16T03:42:09.175068+0000","last_undegraded":"2024-09-16T03:42:09.175068+0000","last_fullsized":"2024-09-16T03:42:09.175068+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'3","last_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_deep_scrub":"100'3","last_deep_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_clean_scrub_stamp":"2024-09-16T03:41:20.484718+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.412321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":254,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:08.792177+0000","last_change":"2024-09-16T03:41:19.857960+0000","last_active":"2024-09-16T03:42:08.792177+0000","last_peered":"2024-09-16T03:42:08.792177+0000","last_clean":"2024-09-16T03:42:08.792177+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:42:08.792177+0000","last_undegraded":"2024-09-16T03:42:08.792177+0000","last_fullsized":"2024-09-16T03:42:08.792177+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:19.857848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:19.857848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:23.649882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":263,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:10.040719+0000","last_change":"2024-09-16T03:42:10.040769+0000","last_active":"2024-09-16T03:42:10.040719+0000","last_peered":"2024-09-16T03:42:10.040719+0000","last_clean":"2024-09-16T03:42:10.040719+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:42:10.040719+0000","last_undegraded":"2024-09-16T03:42:10.040719+0000","last_fullsized":"2024-09-16T03:42:10.040719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:10.040579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:42:10.040579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175076+0000","last_change":"2024-09-16T03:41:55.910523+0000","last_active":"2024-09-16T03:42:09.175076+0000","last_peered":"2024-09-16T03:42:09.175076+0000","last_clean":"2024-09-16T03:42:09.175076+0000","last_became_active":"2024-09-16T03:41:55.910222+0000","last_became_peered":"2024-09-16T03:41:55.910222+0000","last_unstale":"2024-09-16T03:42:09.175076+0000","last_undegraded":"2024-09-16T03:42:09.175076+0000","last_fullsized":"2024-09-16T03:42:09.175076+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_clean_scrub_stamp":"2024-09-16T03:41:07.602473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:28.092968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":252,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175675+0000","last_change":"2024-09-16T03:41:16.470343+0000","last_active":"2024-09-16T03:42:09.175675+0000","last_peered":"2024-09-16T03:42:09.175675+0000","last_clean":"2024-09-16T03:42:09.175675+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:42:09.175675+0000","last_undegraded":"2024-09-16T03:42:09.175675+0000","last_fullsized":"2024-09-16T03:42:09.175675+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:16.470256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:16.470256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:29.234901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,5,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":76,"seq":326417514514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29268,"kb_used_data":2112,"kb_used_omap":3,"kb_used_meta":27132,"kb_avail":94342572,"statfs":{"total":96636764160,"available":96606793728,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3529,"internal_metadata":27783735},"hb_peers":[0,3,4,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":107,"seq":459561500684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29080,"kb_used_data":2112,"kb_used_omap":4,"kb_used_meta":26939,"kb_avail":94342760,"statfs":{"total":96636764160,"available":96606986240,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4385,"internal_metadata":27586271},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29144,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342696,"statfs":{"total":96636764160,"available":96606920704,"internally_reserved":0,"allocated":2166784,"data_stored":1273420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33308,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[3,5,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2166784,"data_stored":1273420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247840,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33824,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338016,"statfs":{"total":96636764160,"available":96602128384,"internally_reserved":0,"allocated":2695168,"data_stored":1802329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,5,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247840,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,5,6,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51872,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319968,"statfs":{"total":96636764160,"available":96583647232,"internally_reserved":0,"allocated":2695168,"data_stored":1802329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:14.299 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T03:42:14.300 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:14.574 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:42:14.574 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:42:14.586 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T03:42:12.940554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":539504,"kb_used_data":34840,"kb_used_omap":29,"kb_used_meta":504290,"kb_avail":1509409936,"statfs":{"total":1546188226560,"available":1545635774464,"internally_reserved":0,"allocated":35676160,"data_stored":21432310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30242,"internal_metadata":516393438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.153433"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":446,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.311209+0000","last_change":"2024-09-16T03:41:29.486608+0000","last_active":"2024-09-16T03:42:09.311209+0000","last_peered":"2024-09-16T03:42:09.311209+0000","last_clean":"2024-09-16T03:42:09.311209+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:42:09.311209+0000","last_undegraded":"2024-09-16T03:42:09.311209+0000","last_fullsized":"2024-09-16T03:42:09.311209+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T03:41:29.486447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:29.486447+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:35.026906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":256,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.311207+0000","last_change":"2024-09-16T03:41:30.473366+0000","last_active":"2024-09-16T03:42:09.311207+0000","last_peered":"2024-09-16T03:42:09.311207+0000","last_clean":"2024-09-16T03:42:09.311207+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:42:09.311207+0000","last_undegraded":"2024-09-16T03:42:09.311207+0000","last_fullsized":"2024-09-16T03:42:09.311207+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'1","last_scrub_stamp":"2024-09-16T03:41:30.473251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:30.473251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:54.704969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":252,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175564+0000","last_change":"2024-09-16T03:41:27.442141+0000","last_active":"2024-09-16T03:42:09.175564+0000","last_peered":"2024-09-16T03:42:09.175564+0000","last_clean":"2024-09-16T03:42:09.175564+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:42:09.175564+0000","last_undegraded":"2024-09-16T03:42:09.175564+0000","last_fullsized":"2024-09-16T03:42:09.175564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.442041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.442041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:42.891354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":253,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175139+0000","last_change":"2024-09-16T03:41:27.654794+0000","last_active":"2024-09-16T03:42:09.175139+0000","last_peered":"2024-09-16T03:42:09.175139+0000","last_clean":"2024-09-16T03:42:09.175139+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:42:09.175139+0000","last_undegraded":"2024-09-16T03:42:09.175139+0000","last_fullsized":"2024-09-16T03:42:09.175139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.654635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.654635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:43.580094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"100'3","reported_seq":215,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175068+0000","last_change":"2024-09-16T03:41:20.484846+0000","last_active":"2024-09-16T03:42:09.175068+0000","last_peered":"2024-09-16T03:42:09.175068+0000","last_clean":"2024-09-16T03:42:09.175068+0000","last_became_active":"2024-09-16T03:40:24.325471+0000","last_became_peered":"2024-09-16T03:40:24.325471+0000","last_unstale":"2024-09-16T03:42:09.175068+0000","last_undegraded":"2024-09-16T03:42:09.175068+0000","last_fullsized":"2024-09-16T03:42:09.175068+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'3","last_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_deep_scrub":"100'3","last_deep_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_clean_scrub_stamp":"2024-09-16T03:41:20.484718+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.412321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":254,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:08.792177+0000","last_change":"2024-09-16T03:41:19.857960+0000","last_active":"2024-09-16T03:42:08.792177+0000","last_peered":"2024-09-16T03:42:08.792177+0000","last_clean":"2024-09-16T03:42:08.792177+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:42:08.792177+0000","last_undegraded":"2024-09-16T03:42:08.792177+0000","last_fullsized":"2024-09-16T03:42:08.792177+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:19.857848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:19.857848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:23.649882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":263,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:10.040719+0000","last_change":"2024-09-16T03:42:10.040769+0000","last_active":"2024-09-16T03:42:10.040719+0000","last_peered":"2024-09-16T03:42:10.040719+0000","last_clean":"2024-09-16T03:42:10.040719+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:42:10.040719+0000","last_undegraded":"2024-09-16T03:42:10.040719+0000","last_fullsized":"2024-09-16T03:42:10.040719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:10.040579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:42:10.040579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175076+0000","last_change":"2024-09-16T03:41:55.910523+0000","last_active":"2024-09-16T03:42:09.175076+0000","last_peered":"2024-09-16T03:42:09.175076+0000","last_clean":"2024-09-16T03:42:09.175076+0000","last_became_active":"2024-09-16T03:41:55.910222+0000","last_became_peered":"2024-09-16T03:41:55.910222+0000","last_unstale":"2024-09-16T03:42:09.175076+0000","last_undegraded":"2024-09-16T03:42:09.175076+0000","last_fullsized":"2024-09-16T03:42:09.175076+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_clean_scrub_stamp":"2024-09-16T03:41:07.602473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:28.092968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":252,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:42:09.175675+0000","last_change":"2024-09-16T03:41:16.470343+0000","last_active":"2024-09-16T03:42:09.175675+0000","last_peered":"2024-09-16T03:42:09.175675+0000","last_clean":"2024-09-16T03:42:09.175675+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:42:09.175675+0000","last_undegraded":"2024-09-16T03:42:09.175675+0000","last_fullsized":"2024-09-16T03:42:09.175675+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:16.470256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:16.470256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:29.234901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,5,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":76,"seq":326417514514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29268,"kb_used_data":2112,"kb_used_omap":3,"kb_used_meta":27132,"kb_avail":94342572,"statfs":{"total":96636764160,"available":96606793728,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3529,"internal_metadata":27783735},"hb_peers":[0,3,4,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":107,"seq":459561500684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29080,"kb_used_data":2112,"kb_used_omap":4,"kb_used_meta":26939,"kb_avail":94342760,"statfs":{"total":96636764160,"available":96606986240,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4385,"internal_metadata":27586271},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29144,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342696,"statfs":{"total":96636764160,"available":96606920704,"internally_reserved":0,"allocated":2166784,"data_stored":1273420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33308,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[3,5,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2166784,"data_stored":1273420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247840,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247838,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33824,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338016,"statfs":{"total":96636764160,"available":96602128384,"internally_reserved":0,"allocated":2695168,"data_stored":1802329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247840,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,5,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247840,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247839,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2162688,"data_stored":1273401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,5,6,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51872,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319968,"statfs":{"total":96636764160,"available":96583647232,"internally_reserved":0,"allocated":2695168,"data_stored":1802329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:14.588 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:42:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:42:14.868 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":143,"fsid":"ecf7a42a-e941-4b24-8693-0ff8b6eb41b1","created":"2024-09-16T03:39:42.686123+0000","modified":"2024-09-16T03:42:12.814685+0000","last_up_change":"2024-09-16T03:41:26.550004+0000","last_in_change":"2024-09-16T03:41:35.577075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:39:56.952065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:39:57.213209+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":27,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0100002288818359,"score_stable":4.0100002288818359,"optimal_score":0.81000000238418579,"raw_score_acting":3.25,"raw_score_stable":3.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8820e1f9-b619-44a5-abb1-7cfa97fcef00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6811","nonce":2834882614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6813","nonce":2834882614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6817","nonce":2834882614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":2834882614},{"type":"v1","addr":"172.21.15.76:6815","nonce":2834882614}]},"public_addr":"172.21.15.76:6811/2834882614","cluster_addr":"172.21.15.76:6813/2834882614","heartbeat_back_addr":"172.21.15.76:6817/2834882614","heartbeat_front_addr":"172.21.15.76:6815/2834882614","state":["exists","up"]},{"osd":1,"uuid":"0f58ae45-e2ff-4c4f-9875-7d34ec312cbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6809","nonce":2045143946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6811","nonce":2045143946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6815","nonce":2045143946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":2045143946},{"type":"v1","addr":"172.21.15.103:6813","nonce":2045143946}]},"public_addr":"172.21.15.103:6809/2045143946","cluster_addr":"172.21.15.103:6811/2045143946","heartbeat_back_addr":"172.21.15.103:6815/2045143946","heartbeat_front_addr":"172.21.15.103:6813/2045143946","state":["exists","up"]},{"osd":2,"uuid":"b369fe92-86b8-45b3-a418-024f0e4becd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6817","nonce":273257286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6819","nonce":273257286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6823","nonce":273257286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":273257286},{"type":"v1","addr":"172.21.15.119:6821","nonce":273257286}]},"public_addr":"172.21.15.119:6817/273257286","cluster_addr":"172.21.15.119:6819/273257286","heartbeat_back_addr":"172.21.15.119:6823/273257286","heartbeat_front_addr":"172.21.15.119:6821/273257286","state":["exists","up"]},{"osd":3,"uuid":"b9521bf1-46ef-402c-8472-c1b476280b23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6811","nonce":278480145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6815","nonce":278480145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6821","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6823","nonce":278480145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":278480145},{"type":"v1","addr":"172.21.15.121:6819","nonce":278480145}]},"public_addr":"172.21.15.121:6811/278480145","cluster_addr":"172.21.15.121:6815/278480145","heartbeat_back_addr":"172.21.15.121:6823/278480145","heartbeat_front_addr":"172.21.15.121:6819/278480145","state":["exists","up"]},{"osd":4,"uuid":"15bec6a1-fc13-4822-8cfa-899b83886c10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6803","nonce":3302785479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6805","nonce":3302785479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6809","nonce":3302785479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":3302785479},{"type":"v1","addr":"172.21.15.76:6807","nonce":3302785479}]},"public_addr":"172.21.15.76:6803/3302785479","cluster_addr":"172.21.15.76:6805/3302785479","heartbeat_back_addr":"172.21.15.76:6809/3302785479","heartbeat_front_addr":"172.21.15.76:6807/3302785479","state":["exists","up"]},{"osd":5,"uuid":"e9b234ec-bfa4-4f72-adab-295e067b2e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6801","nonce":3765230361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6803","nonce":3765230361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6807","nonce":3765230361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3765230361},{"type":"v1","addr":"172.21.15.103:6805","nonce":3765230361}]},"public_addr":"172.21.15.103:6801/3765230361","cluster_addr":"172.21.15.103:6803/3765230361","heartbeat_back_addr":"172.21.15.103:6807/3765230361","heartbeat_front_addr":"172.21.15.103:6805/3765230361","state":["exists","up"]},{"osd":6,"uuid":"6d4ae9fb-8d8f-451d-a12f-1265302b03ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6825","nonce":446165121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6827","nonce":446165121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6831","nonce":446165121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":446165121},{"type":"v1","addr":"172.21.15.119:6829","nonce":446165121}]},"public_addr":"172.21.15.119:6825/446165121","cluster_addr":"172.21.15.119:6827/446165121","heartbeat_back_addr":"172.21.15.119:6831/446165121","heartbeat_front_addr":"172.21.15.119:6829/446165121","state":["exists","up"]},{"osd":7,"uuid":"2c9456ed-00ec-46aa-814e-f85eb441fc4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6810","nonce":1304791608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6814","nonce":1304791608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6822","nonce":1304791608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1304791608},{"type":"v1","addr":"172.21.15.121:6818","nonce":1304791608}]},"public_addr":"172.21.15.121:6810/1304791608","cluster_addr":"172.21.15.121:6814/1304791608","heartbeat_back_addr":"172.21.15.121:6822/1304791608","heartbeat_front_addr":"172.21.15.121:6818/1304791608","state":["exists","up"]},{"osd":8,"uuid":"b618a02a-8628-48f5-bb5f-f7591aafa118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6819","nonce":2630853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6821","nonce":2630853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6825","nonce":2630853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":2630853906},{"type":"v1","addr":"172.21.15.76:6823","nonce":2630853906}]},"public_addr":"172.21.15.76:6819/2630853906","cluster_addr":"172.21.15.76:6821/2630853906","heartbeat_back_addr":"172.21.15.76:6825/2630853906","heartbeat_front_addr":"172.21.15.76:6823/2630853906","state":["exists","up"]},{"osd":9,"uuid":"dc3e5be7-f012-47e6-ba35-0e3354ca0c18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6817","nonce":2702548675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6819","nonce":2702548675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6823","nonce":2702548675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2702548675},{"type":"v1","addr":"172.21.15.103:6821","nonce":2702548675}]},"public_addr":"172.21.15.103:6817/2702548675","cluster_addr":"172.21.15.103:6819/2702548675","heartbeat_back_addr":"172.21.15.103:6823/2702548675","heartbeat_front_addr":"172.21.15.103:6821/2702548675","state":["exists","up"]},{"osd":10,"uuid":"4e70a7a8-7ba8-42ab-a45a-e0564b561619","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6809","nonce":205142394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6811","nonce":205142394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6815","nonce":205142394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":205142394},{"type":"v1","addr":"172.21.15.119:6813","nonce":205142394}]},"public_addr":"172.21.15.119:6809/205142394","cluster_addr":"172.21.15.119:6811/205142394","heartbeat_back_addr":"172.21.15.119:6815/205142394","heartbeat_front_addr":"172.21.15.119:6813/205142394","state":["exists","up"]},{"osd":11,"uuid":"d16bc493-335f-41c2-bd68-cbad93c49a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6825","nonce":1551091359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6827","nonce":1551091359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6831","nonce":1551091359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1551091359},{"type":"v1","addr":"172.21.15.121:6829","nonce":1551091359}]},"public_addr":"172.21.15.121:6825/1551091359","cluster_addr":"172.21.15.121:6827/1551091359","heartbeat_back_addr":"172.21.15.121:6831/1551091359","heartbeat_front_addr":"172.21.15.121:6829/1551091359","state":["exists","up"]},{"osd":12,"uuid":"a8e6ffb2-04c0-4a43-90ec-f4da8bb7e1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":83,"up_from":107,"up_thru":55,"down_at":84,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6827","nonce":217312017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6829","nonce":217312017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6833","nonce":217312017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":217312017},{"type":"v1","addr":"172.21.15.76:6831","nonce":217312017}]},"public_addr":"172.21.15.76:6827/217312017","cluster_addr":"172.21.15.76:6829/217312017","heartbeat_back_addr":"172.21.15.76:6833/217312017","heartbeat_front_addr":"172.21.15.76:6831/217312017","state":["exists","up"]},{"osd":13,"uuid":"61e9204b-8269-4c35-b409-5cfbd5e6abbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":53,"up_from":76,"up_thru":110,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6825","nonce":3364003535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6827","nonce":3364003535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6831","nonce":3364003535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3364003535},{"type":"v1","addr":"172.21.15.103:6829","nonce":3364003535}]},"public_addr":"172.21.15.103:6825/3364003535","cluster_addr":"172.21.15.103:6827/3364003535","heartbeat_back_addr":"172.21.15.103:6831/3364003535","heartbeat_front_addr":"172.21.15.103:6829/3364003535","state":["exists","up"]},{"osd":14,"uuid":"0bd5338e-eb99-44cc-9f5f-1c8ef903d02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6801","nonce":1147733023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6803","nonce":1147733023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6807","nonce":1147733023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1147733023},{"type":"v1","addr":"172.21.15.119:6805","nonce":1147733023}]},"public_addr":"172.21.15.119:6801/1147733023","cluster_addr":"172.21.15.119:6803/1147733023","heartbeat_back_addr":"172.21.15.119:6807/1147733023","heartbeat_front_addr":"172.21.15.119:6805/1147733023","state":["exists","up"]},{"osd":15,"uuid":"f240405e-d268-41cc-bd99-5a2ca71a85a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6801","nonce":3149764080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6803","nonce":3149764080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6807","nonce":3149764080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3149764080},{"type":"v1","addr":"172.21.15.121:6805","nonce":3149764080}]},"public_addr":"172.21.15.121:6801/3149764080","cluster_addr":"172.21.15.121:6803/3149764080","heartbeat_back_addr":"172.21.15.121:6807/3149764080","heartbeat_front_addr":"172.21.15.121:6805/3149764080","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:11.911108+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:12.029403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:21.537323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:59.887564+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:01.297243+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:00.164464+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.217756+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:58.131838+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:03.346738+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:11.711895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:19.361985+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:24.073827+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:25.680752+0000","dead_epoch":84},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:11.065521+0000","dead_epoch":54},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:09.217323+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:41:07.187965+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.2","osds":[14,3]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":11,"length":1},{"begin":33,"length":1},{"begin":38,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:42:15.881 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T03:42:15.882 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:16.015 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:16.015 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:16.015 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:16.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T03:42:16.303 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T03:42:16.316 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T03:42:16.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:16.440 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:16.440 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:16.440 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:16.451 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T03:42:16.728 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T03:42:16.741 INFO:tasks.ceph:Scrubbing osd.8 2024-09-16T03:42:16.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:16.865 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:16.865 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:16.865 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:16.876 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-16T03:42:17.154 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 8 to deep-scrub 2024-09-16T03:42:17.167 INFO:tasks.ceph:Scrubbing osd.12 2024-09-16T03:42:17.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:17.290 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:17.291 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-16T03:42:17.291 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:17.301 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-16T03:42:17.580 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 12 to deep-scrub 2024-09-16T03:42:17.593 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T03:42:17.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:17.717 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:17.717 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:17.728 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T03:42:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T03:42:18.018 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T03:42:18.019 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:18.144 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:18.144 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:18.154 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T03:42:18.431 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T03:42:18.444 INFO:tasks.ceph:Scrubbing osd.9 2024-09-16T03:42:18.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:18.570 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:18.570 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:18.570 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:18.580 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-16T03:42:18.860 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 9 to deep-scrub 2024-09-16T03:42:18.872 INFO:tasks.ceph:Scrubbing osd.13 2024-09-16T03:42:18.873 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:19.001 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:19.002 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:19.002 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:19.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-16T03:42:19.291 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 13 to deep-scrub 2024-09-16T03:42:19.304 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T03:42:19.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:19.428 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:19.428 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:19.428 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:19.439 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T03:42:19.716 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T03:42:19.730 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T03:42:19.730 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:19.855 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:19.855 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:19.866 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T03:42:20.145 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T03:42:20.158 INFO:tasks.ceph:Scrubbing osd.10 2024-09-16T03:42:20.158 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:20.283 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:20.283 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:20.283 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:20.294 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-16T03:42:20.574 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 10 to deep-scrub 2024-09-16T03:42:20.587 INFO:tasks.ceph:Scrubbing osd.14 2024-09-16T03:42:20.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:20.713 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:20.713 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:20.713 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:20.723 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-16T03:42:21.004 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 14 to deep-scrub 2024-09-16T03:42:21.018 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T03:42:21.018 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:21.143 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:21.143 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:21.144 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:21.154 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T03:42:21.431 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T03:42:21.444 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T03:42:21.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:21.568 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:21.568 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:21.568 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:21.579 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T03:42:21.860 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T03:42:21.873 INFO:tasks.ceph:Scrubbing osd.11 2024-09-16T03:42:21.873 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:21.998 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:21.998 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:21.998 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:22.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-16T03:42:22.290 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 11 to deep-scrub 2024-09-16T03:42:22.303 INFO:tasks.ceph:Scrubbing osd.15 2024-09-16T03:42:22.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:42:22.428 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T03:42:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:42:22.429 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T03:42:22.439 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-16T03:42:22.718 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 15 to deep-scrub 2024-09-16T03:42:22.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:23.007 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:42:23.007 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:42:23.019 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-16T03:42:22.942372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":540072,"kb_used_data":35432,"kb_used_omap":29,"kb_used_meta":504290,"kb_avail":1509409368,"statfs":{"total":1546188226560,"available":1545635192832,"internally_reserved":0,"allocated":36282368,"data_stored":21790803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30242,"internal_metadata":516393438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001467"},"pg_stats":[{"pgid":"1.0","version":"26'157","reported_seq":452,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-16T03:42:21.533110+0000","last_change":"2024-09-16T03:41:29.486608+0000","last_active":"2024-09-16T03:42:21.533110+0000","last_peered":"2024-09-16T03:42:21.533110+0000","last_clean":"2024-09-16T03:42:21.533110+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:42:21.533110+0000","last_undegraded":"2024-09-16T03:42:21.533110+0000","last_fullsized":"2024-09-16T03:42:21.533110+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2024-09-16T03:41:29.486447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:29.486447+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:35.026906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":262,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-16T03:42:21.533121+0000","last_change":"2024-09-16T03:41:30.473366+0000","last_active":"2024-09-16T03:42:21.533121+0000","last_peered":"2024-09-16T03:42:21.533121+0000","last_clean":"2024-09-16T03:42:21.533121+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:42:21.533121+0000","last_undegraded":"2024-09-16T03:42:21.533121+0000","last_fullsized":"2024-09-16T03:42:21.533121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'1","last_scrub_stamp":"2024-09-16T03:41:30.473251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:30.473251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:54.704969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":267,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-16T03:42:21.532975+0000","last_change":"2024-09-16T03:42:17.467985+0000","last_active":"2024-09-16T03:42:21.532975+0000","last_peered":"2024-09-16T03:42:21.532975+0000","last_clean":"2024-09-16T03:42:21.532975+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:42:21.532975+0000","last_undegraded":"2024-09-16T03:42:21.532975+0000","last_fullsized":"2024-09-16T03:42:21.532975+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:17.467888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:17.467888+0000","last_clean_scrub_stamp":"2024-09-16T03:42:17.467888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:33.557763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":259,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-16T03:42:21.533298+0000","last_change":"2024-09-16T03:41:27.654794+0000","last_active":"2024-09-16T03:42:21.533298+0000","last_peered":"2024-09-16T03:42:21.533298+0000","last_clean":"2024-09-16T03:42:21.533298+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:42:21.533298+0000","last_undegraded":"2024-09-16T03:42:21.533298+0000","last_fullsized":"2024-09-16T03:42:21.533298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:27.654635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:57.247142+0000","last_clean_scrub_stamp":"2024-09-16T03:41:27.654635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:43.580094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"100'3","reported_seq":219,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T03:42:18.499345+0000","last_change":"2024-09-16T03:41:20.484846+0000","last_active":"2024-09-16T03:42:18.499345+0000","last_peered":"2024-09-16T03:42:18.499345+0000","last_clean":"2024-09-16T03:42:18.499345+0000","last_became_active":"2024-09-16T03:40:24.325471+0000","last_became_peered":"2024-09-16T03:40:24.325471+0000","last_unstale":"2024-09-16T03:42:18.499345+0000","last_undegraded":"2024-09-16T03:42:18.499345+0000","last_fullsized":"2024-09-16T03:42:18.499345+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'3","last_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_deep_scrub":"100'3","last_deep_scrub_stamp":"2024-09-16T03:41:20.484718+0000","last_clean_scrub_stamp":"2024-09-16T03:41:20.484718+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.412321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":267,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T03:42:20.551251+0000","last_change":"2024-09-16T03:42:20.551292+0000","last_active":"2024-09-16T03:42:20.551251+0000","last_peered":"2024-09-16T03:42:20.551251+0000","last_clean":"2024-09-16T03:42:20.551251+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:42:20.551251+0000","last_undegraded":"2024-09-16T03:42:20.551251+0000","last_fullsized":"2024-09-16T03:42:20.551251+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:20.551149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:20.551149+0000","last_clean_scrub_stamp":"2024-09-16T03:42:20.551149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:33.724213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":278,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-16T03:42:20.862775+0000","last_change":"2024-09-16T03:42:18.978609+0000","last_active":"2024-09-16T03:42:20.862775+0000","last_peered":"2024-09-16T03:42:20.862775+0000","last_clean":"2024-09-16T03:42:20.862775+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:42:20.862775+0000","last_undegraded":"2024-09-16T03:42:20.862775+0000","last_fullsized":"2024-09-16T03:42:20.862775+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:18.978488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:18.978488+0000","last_clean_scrub_stamp":"2024-09-16T03:42:18.978488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:43.929280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":36,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T03:42:18.499342+0000","last_change":"2024-09-16T03:41:55.910523+0000","last_active":"2024-09-16T03:42:18.499342+0000","last_peered":"2024-09-16T03:42:18.499342+0000","last_clean":"2024-09-16T03:42:18.499342+0000","last_became_active":"2024-09-16T03:41:55.910222+0000","last_became_peered":"2024-09-16T03:41:55.910222+0000","last_unstale":"2024-09-16T03:42:18.499342+0000","last_undegraded":"2024-09-16T03:42:18.499342+0000","last_fullsized":"2024-09-16T03:42:18.499342+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:07.602473+0000","last_clean_scrub_stamp":"2024-09-16T03:41:07.602473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:28.092968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":267,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-16T03:42:21.532969+0000","last_change":"2024-09-16T03:42:18.474803+0000","last_active":"2024-09-16T03:42:21.532969+0000","last_peered":"2024-09-16T03:42:21.532969+0000","last_clean":"2024-09-16T03:42:21.532969+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:42:21.532969+0000","last_undegraded":"2024-09-16T03:42:21.532969+0000","last_fullsized":"2024-09-16T03:42:21.532969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:18.474710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:18.474710+0000","last_clean_scrub_stamp":"2024-09-16T03:42:18.474710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:39.615179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33348,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338492,"statfs":{"total":96636764160,"available":96602615808,"internally_reserved":0,"allocated":2211840,"data_stored":1302468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,5,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":76,"seq":326417514516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29300,"kb_used_data":2144,"kb_used_omap":3,"kb_used_meta":27132,"kb_avail":94342540,"statfs":{"total":96636764160,"available":96606760960,"internally_reserved":0,"allocated":2195456,"data_stored":1292779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3529,"internal_metadata":27783735},"hb_peers":[0,3,4,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":107,"seq":459561500686,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29108,"kb_used_data":2144,"kb_used_omap":4,"kb_used_meta":26939,"kb_avail":94342732,"statfs":{"total":96636764160,"available":96606957568,"internally_reserved":0,"allocated":2195456,"data_stored":1292779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4385,"internal_metadata":27586271},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313249,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29176,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342664,"statfs":{"total":96636764160,"available":96606887936,"internally_reserved":0,"allocated":2199552,"data_stored":1292798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33332,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338508,"statfs":{"total":96636764160,"available":96602632192,"internally_reserved":0,"allocated":2195456,"data_stored":1292779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33332,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338508,"statfs":{"total":96636764160,"available":96602632192,"internally_reserved":0,"allocated":2195456,"data_stored":1292779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33332,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338508,"statfs":{"total":96636764160,"available":96602632192,"internally_reserved":0,"allocated":2195456,"data_stored":1292779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[3,5,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2199552,"data_stored":1292798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33332,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338508,"statfs":{"total":96636764160,"available":96602632192,"internally_reserved":0,"allocated":2195456,"data_stored":1292779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33348,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338492,"statfs":{"total":96636764160,"available":96602615808,"internally_reserved":0,"allocated":2211840,"data_stored":1302468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247842,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33348,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338492,"statfs":{"total":96636764160,"available":96602615808,"internally_reserved":0,"allocated":2211840,"data_stored":1302468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247840,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33852,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337988,"statfs":{"total":96636764160,"available":96602099712,"internally_reserved":0,"allocated":2727936,"data_stored":1821707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33332,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338508,"statfs":{"total":96636764160,"available":96602632192,"internally_reserved":0,"allocated":2195456,"data_stored":1292779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,5,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247842,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33332,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338508,"statfs":{"total":96636764160,"available":96602632192,"internally_reserved":0,"allocated":2195456,"data_stored":1292779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247841,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33348,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338492,"statfs":{"total":96636764160,"available":96602615808,"internally_reserved":0,"allocated":2211840,"data_stored":1302468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,5,6,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247841,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51916,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319924,"statfs":{"total":96636764160,"available":96583602176,"internally_reserved":0,"allocated":2744320,"data_stored":1831396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:23.025 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T03:41:29.486447+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:42:23.025 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T03:41:30.473251+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:42:23.025 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T03:41:27.654635+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:42:23.026 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T03:41:20.484718+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=20, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:42:23.026 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T03:41:07.602473+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=7, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:42:23.026 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T03:42:43.027 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:43.323 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T03:42:43.324 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T03:42:43.336 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-16T03:42:42.947140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164,"ondisk_log_size":164,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":546184,"kb_used_data":37352,"kb_used_omap":29,"kb_used_meta":508450,"kb_avail":1509403256,"statfs":{"total":1546188226560,"available":1545628934144,"internally_reserved":0,"allocated":38248448,"data_stored":22954283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30242,"internal_metadata":520653278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001455"},"pg_stats":[{"pgid":"1.0","version":"145'159","reported_seq":477,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T03:42:40.559706+0000","last_change":"2024-09-16T03:42:22.690777+0000","last_active":"2024-09-16T03:42:40.559706+0000","last_peered":"2024-09-16T03:42:40.559706+0000","last_clean":"2024-09-16T03:42:40.559706+0000","last_became_active":"2024-09-16T03:39:58.261453+0000","last_became_peered":"2024-09-16T03:39:58.261453+0000","last_unstale":"2024-09-16T03:42:40.559706+0000","last_undegraded":"2024-09-16T03:42:40.559706+0000","last_fullsized":"2024-09-16T03:42:40.559706+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'159","last_scrub_stamp":"2024-09-16T03:42:22.690654+0000","last_deep_scrub":"145'159","last_deep_scrub_stamp":"2024-09-16T03:42:22.690654+0000","last_clean_scrub_stamp":"2024-09-16T03:42:22.690654+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:32.483554+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"145'2","reported_seq":286,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T03:42:40.559746+0000","last_change":"2024-09-16T03:42:23.657963+0000","last_active":"2024-09-16T03:42:40.559746+0000","last_peered":"2024-09-16T03:42:40.559746+0000","last_clean":"2024-09-16T03:42:40.559746+0000","last_became_active":"2024-09-16T03:39:58.852349+0000","last_became_peered":"2024-09-16T03:39:58.852349+0000","last_unstale":"2024-09-16T03:42:40.559746+0000","last_undegraded":"2024-09-16T03:42:40.559746+0000","last_fullsized":"2024-09-16T03:42:40.559746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'2","last_scrub_stamp":"2024-09-16T03:42:23.657811+0000","last_deep_scrub":"145'2","last_deep_scrub_stamp":"2024-09-16T03:42:23.657811+0000","last_clean_scrub_stamp":"2024-09-16T03:42:23.657811+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:38.936452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046268600000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":281,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T03:42:40.185319+0000","last_change":"2024-09-16T03:42:17.467985+0000","last_active":"2024-09-16T03:42:40.185319+0000","last_peered":"2024-09-16T03:42:40.185319+0000","last_clean":"2024-09-16T03:42:40.185319+0000","last_became_active":"2024-09-16T03:39:58.266985+0000","last_became_peered":"2024-09-16T03:39:58.266985+0000","last_unstale":"2024-09-16T03:42:40.185319+0000","last_undegraded":"2024-09-16T03:42:40.185319+0000","last_fullsized":"2024-09-16T03:42:40.185319+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:17.467888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:17.467888+0000","last_clean_scrub_stamp":"2024-09-16T03:42:17.467888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:33.557763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050354499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":281,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T03:42:40.716903+0000","last_change":"2024-09-16T03:42:22.736070+0000","last_active":"2024-09-16T03:42:40.716903+0000","last_peered":"2024-09-16T03:42:40.716903+0000","last_clean":"2024-09-16T03:42:40.716903+0000","last_became_active":"2024-09-16T03:39:58.261952+0000","last_became_peered":"2024-09-16T03:39:58.261952+0000","last_unstale":"2024-09-16T03:42:40.716903+0000","last_undegraded":"2024-09-16T03:42:40.716903+0000","last_fullsized":"2024-09-16T03:42:40.716903+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:22.735898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:22.735898+0000","last_clean_scrub_stamp":"2024-09-16T03:42:22.735898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:28.729224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000402598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"100'3","reported_seq":243,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T03:42:39.000666+0000","last_change":"2024-09-16T03:42:22.119997+0000","last_active":"2024-09-16T03:42:39.000666+0000","last_peered":"2024-09-16T03:42:39.000666+0000","last_clean":"2024-09-16T03:42:39.000666+0000","last_became_active":"2024-09-16T03:40:24.325471+0000","last_became_peered":"2024-09-16T03:40:24.325471+0000","last_unstale":"2024-09-16T03:42:39.000666+0000","last_undegraded":"2024-09-16T03:42:39.000666+0000","last_fullsized":"2024-09-16T03:42:39.000666+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'3","last_scrub_stamp":"2024-09-16T03:42:22.119867+0000","last_deep_scrub":"100'3","last_deep_scrub_stamp":"2024-09-16T03:42:22.119867+0000","last_clean_scrub_stamp":"2024-09-16T03:42:22.119867+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:42.980385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073518599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":283,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T03:42:40.559426+0000","last_change":"2024-09-16T03:42:20.551292+0000","last_active":"2024-09-16T03:42:40.559426+0000","last_peered":"2024-09-16T03:42:40.559426+0000","last_clean":"2024-09-16T03:42:40.559426+0000","last_became_active":"2024-09-16T03:39:58.535649+0000","last_became_peered":"2024-09-16T03:39:58.535649+0000","last_unstale":"2024-09-16T03:42:40.559426+0000","last_undegraded":"2024-09-16T03:42:40.559426+0000","last_fullsized":"2024-09-16T03:42:40.559426+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:20.551149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:20.551149+0000","last_clean_scrub_stamp":"2024-09-16T03:42:20.551149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:33.724213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034653800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":292,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T03:42:40.559941+0000","last_change":"2024-09-16T03:42:18.978609+0000","last_active":"2024-09-16T03:42:40.559941+0000","last_peered":"2024-09-16T03:42:40.559941+0000","last_clean":"2024-09-16T03:42:40.559941+0000","last_became_active":"2024-09-16T03:39:58.265690+0000","last_became_peered":"2024-09-16T03:39:58.265690+0000","last_unstale":"2024-09-16T03:42:40.559941+0000","last_undegraded":"2024-09-16T03:42:40.559941+0000","last_fullsized":"2024-09-16T03:42:40.559941+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:18.978488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:18.978488+0000","last_clean_scrub_stamp":"2024-09-16T03:42:18.978488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:43.929280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041297600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T03:42:39.000664+0000","last_change":"2024-09-16T03:42:23.094543+0000","last_active":"2024-09-16T03:42:39.000664+0000","last_peered":"2024-09-16T03:42:39.000664+0000","last_clean":"2024-09-16T03:42:39.000664+0000","last_became_active":"2024-09-16T03:41:55.910222+0000","last_became_peered":"2024-09-16T03:41:55.910222+0000","last_unstale":"2024-09-16T03:42:39.000664+0000","last_undegraded":"2024-09-16T03:42:39.000664+0000","last_fullsized":"2024-09-16T03:42:39.000664+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:23.094423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:23.094423+0000","last_clean_scrub_stamp":"2024-09-16T03:42:23.094423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:31.144799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":281,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T03:42:40.185321+0000","last_change":"2024-09-16T03:42:18.474803+0000","last_active":"2024-09-16T03:42:40.185321+0000","last_peered":"2024-09-16T03:42:40.185321+0000","last_clean":"2024-09-16T03:42:40.185321+0000","last_became_active":"2024-09-16T03:39:58.368875+0000","last_became_peered":"2024-09-16T03:39:58.368875+0000","last_unstale":"2024-09-16T03:42:40.185321+0000","last_undegraded":"2024-09-16T03:42:40.185321+0000","last_fullsized":"2024-09-16T03:42:40.185321+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:18.474710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:18.474710+0000","last_clean_scrub_stamp":"2024-09-16T03:42:18.474710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:39.615179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00030821400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2326528,"data_stored":1370341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,5,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":76,"seq":326417514520,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29428,"kb_used_data":2272,"kb_used_omap":3,"kb_used_meta":27132,"kb_avail":94342412,"statfs":{"total":96636764160,"available":96606629888,"internally_reserved":0,"allocated":2326528,"data_stored":1370341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3529,"internal_metadata":27783735},"hb_peers":[0,3,4,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":107,"seq":459561500690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29236,"kb_used_data":2272,"kb_used_omap":4,"kb_used_meta":26939,"kb_avail":94342604,"statfs":{"total":96636764160,"available":96606826496,"internally_reserved":0,"allocated":2326528,"data_stored":1370341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4385,"internal_metadata":27586271},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313253,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33452,"kb_used_data":2260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338388,"statfs":{"total":96636764160,"available":96602509312,"internally_reserved":0,"allocated":2314240,"data_stored":1360671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280549,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":2310144,"data_stored":1360652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280549,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":2310144,"data_stored":1360652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280548,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33464,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2326528,"data_stored":1370341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[3,5,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33468,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338372,"statfs":{"total":96636764160,"available":96602492928,"internally_reserved":0,"allocated":2330624,"data_stored":1370360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2326528,"data_stored":1370341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2326528,"data_stored":1370341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247846,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33460,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2326528,"data_stored":1370341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247844,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33980,"kb_used_data":2792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337860,"statfs":{"total":96636764160,"available":96601968640,"internally_reserved":0,"allocated":2859008,"data_stored":1899269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247846,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2326528,"data_stored":1370341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,5,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247846,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2326528,"data_stored":1370341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2326528,"data_stored":1370341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,5,6,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247845,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52032,"kb_used_data":2792,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319808,"statfs":{"total":96636764160,"available":96583483392,"internally_reserved":0,"allocated":2859008,"data_stored":1899269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:43.339 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T03:42:43.635 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T03:42:43.635 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T03:42:43.635 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T03:42:43.636 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:43.665 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T03:42:43.666 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T03:42:43.666 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:43.746 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T03:42:43.747 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-16T03:42:43.747 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:43.818 INFO:tasks.ceph.osd.8:Stopped 2024-09-16T03:42:43.819 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-16T03:42:43.819 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:43.891 INFO:tasks.ceph.osd.12:Stopped 2024-09-16T03:42:43.891 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T03:42:43.891 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:43.933 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T03:42:43.933 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T03:42:43.934 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:43.976 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T03:42:43.977 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-16T03:42:43.977 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.018 INFO:tasks.ceph.osd.9:Stopped 2024-09-16T03:42:44.018 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-16T03:42:44.018 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.053 INFO:tasks.ceph.osd.13:Stopped 2024-09-16T03:42:44.053 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T03:42:44.053 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.090 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T03:42:44.090 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T03:42:44.090 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.164 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T03:42:44.165 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-16T03:42:44.165 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.235 INFO:tasks.ceph.osd.10:Stopped 2024-09-16T03:42:44.235 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-16T03:42:44.235 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.303 INFO:tasks.ceph.osd.14:Stopped 2024-09-16T03:42:44.303 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T03:42:44.303 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.335 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T03:42:44.335 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T03:42:44.335 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.404 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T03:42:44.404 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-16T03:42:44.404 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.476 INFO:tasks.ceph.osd.11:Stopped 2024-09-16T03:42:44.477 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-16T03:42:44.477 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.552 INFO:tasks.ceph.osd.15:Stopped 2024-09-16T03:42:44.552 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T03:42:44.553 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T03:42:44.553 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.579 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T03:42:44.579 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T03:42:44.580 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.622 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T03:42:44.622 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T03:42:44.622 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T03:42:44.622 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.634 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T03:42:44.634 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:42:44.634 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.648 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:42:44.648 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:42:44.648 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:44.662 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:42:44.662 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T03:42:44.663 DEBUG:teuthology.orchestra.run.smithi076:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-16T03:42:44.714 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:42:44.715 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T03:42:44.901 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:42:44.901 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T03:42:44.996 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:42:44.996 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-16T03:42:45.053 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:42:45.053 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-16T03:42:45.120 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:42:45.121 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T03:42:45.259 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:42:45.259 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T03:42:45.339 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:42:45.339 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-16T03:42:45.397 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:42:45.397 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-16T03:42:45.453 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:42:45.453 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T03:42:45.625 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:42:45.625 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T03:42:45.722 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:42:45.722 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-16T03:42:45.781 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:42:45.781 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-16T03:42:45.839 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:42:45.839 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T03:42:46.154 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:42:46.154 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T03:42:46.234 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:42:46.234 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-16T03:42:46.284 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:42:46.284 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-16T03:42:46.334 INFO:tasks.ceph:Archiving mon data... 2024-09-16T03:42:46.335 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/data/mon.a.tgz 2024-09-16T03:42:46.335 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-09-16T03:42:46.354 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.5TdL4Loo7c 2024-09-16T03:42:46.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.5TdL4Loo7c 2024-09-16T03:42:46.530 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.5TdL4Loo7c 2024-09-16T03:42:46.615 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.5TdL4Loo7c is 433KB 2024-09-16T03:42:46.716 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.5TdL4Loo7c 2024-09-16T03:42:46.731 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/data/mon.b.tgz 2024-09-16T03:42:46.731 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-09-16T03:42:46.750 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.yYedqRhWyt 2024-09-16T03:42:46.750 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.yYedqRhWyt 2024-09-16T03:42:47.182 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.yYedqRhWyt 2024-09-16T03:42:47.267 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.yYedqRhWyt is 608KB 2024-09-16T03:42:47.338 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.yYedqRhWyt 2024-09-16T03:42:47.353 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/data/mon.c.tgz 2024-09-16T03:42:47.354 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-09-16T03:42:47.373 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.33qrqII8OY 2024-09-16T03:42:47.373 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.33qrqII8OY 2024-09-16T03:42:47.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.33qrqII8OY 2024-09-16T03:42:47.672 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.33qrqII8OY is 608KB 2024-09-16T03:42:47.774 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.33qrqII8OY 2024-09-16T03:42:47.789 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T03:42:47.789 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T03:42:47.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T03:42:47.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T03:42:47.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T03:42:47.878 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T03:42:47.893 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T03:42:47.909 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T03:42:47.926 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T03:42:47.927 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T03:42:47.928 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi076/crash 2024-09-16T03:42:47.929 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T03:42:47.957 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi103/crash 2024-09-16T03:42:47.958 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T03:42:47.983 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi119/crash 2024-09-16T03:42:47.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T03:42:48.008 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi121/crash 2024-09-16T03:42:48.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T03:42:48.091 INFO:tasks.ceph:Compressing logs... 2024-09-16T03:42:48.091 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T03:42:48.095 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T03:42:48.097 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T03:42:48.099 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T03:42:48.123 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T03:42:48.123 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-16T03:42:48.124 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.39221.log 2024-09-16T03:42:48.124 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-16T03:42:48.124 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/objectstore_tool.39221.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.43092.log 2024-09-16T03:42:48.124 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.42682.log 2024-09-16T03:42:48.124 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.13.log: /var/log/ceph/bluestore_tool.43092.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T03:42:48.124 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/bluestore_tool.42682.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.42703.log 2024-09-16T03:42:48.124 INFO:teuthology.orchestra.run.smithi103.stderr: 85.2% -- replaced with /var/log/ceph/bluestore_tool.42682.log.gz 2024-09-16T03:42:48.124 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.41708.log 2024-09-16T03:42:48.124 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/bluestore_tool.42703.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.40939.log 2024-09-16T03:42:48.124 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/objectstore_tool.41708.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T03:42:48.129 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-16T03:42:48.129 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T03:42:48.129 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T03:42:48.129 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.10.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T03:42:48.130 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-16T03:42:48.130 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T03:42:48.130 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T03:42:48.130 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.6.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T03:42:48.130 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.log: 86.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T03:42:48.145 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/bluestore_tool.42703.log.gz 2024-09-16T03:42:48.145 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/bluestore_tool.43092.log.gz 2024-09-16T03:42:48.145 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T03:42:48.145 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/objectstore_tool.40939.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T03:42:48.145 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.40170.log 2024-09-16T03:42:48.145 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.log: 86.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T03:42:48.146 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T03:42:48.146 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/objectstore_tool.40170.log: 93.2% -- replaced with /var/log/ceph/objectstore_tool.41708.log.gz 2024-09-16T03:42:48.146 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.43481.log 2024-09-16T03:42:48.153 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33060.log 2024-09-16T03:42:48.153 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T03:42:48.153 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.33060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33020.log 2024-09-16T03:42:48.153 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33060.log.gz 2024-09-16T03:42:48.153 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32529.log 2024-09-16T03:42:48.153 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.33020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38290.log 2024-09-16T03:42:48.153 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33020.log.gz 2024-09-16T03:42:48.153 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32529.log.gz 2024-09-16T03:42:48.153 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32820.log 2024-09-16T03:42:48.153 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.0.38290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32740.log 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr: 78.0% -- replaced with /var/log/ceph/ceph-client.0.38290.log.gz 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32820.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32820.log.gz 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32660.log 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32740.log.gz 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32614.log 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33100.log 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32660.log.gz 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32614.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-osd.11.log 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32614.log.gz 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.33100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33100.log.gz 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32569.log 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32860.log 2024-09-16T03:42:48.154 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32569.log.gz 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32940.log 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32860.log.gz 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32780.log 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32489.log 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32940.log.gz 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32900.log 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32780.log.gz 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32700.log 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32489.log.gz 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32900.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32900.log.gz 2024-09-16T03:42:48.155 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32980.log 2024-09-16T03:42:48.156 INFO:teuthology.orchestra.run.smithi121.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32700.log.gz 2024-09-16T03:42:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.32980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32980.log.gz 2024-09-16T03:42:48.156 INFO:teuthology.orchestra.run.smithi121.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T03:42:48.160 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.b.log: 93.3% -- replaced with /var/log/ceph/objectstore_tool.40939.log.gz 2024-09-16T03:42:48.160 INFO:teuthology.orchestra.run.smithi103.stderr: 93.4% -- replaced with /var/log/ceph/objectstore_tool.40170.log.gz 2024-09-16T03:42:48.160 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/bluestore_tool.43481.log: 94.1% -- replaced with /var/log/ceph/objectstore_tool.39221.log.gz 2024-09-16T03:42:48.164 INFO:teuthology.orchestra.run.smithi103.stderr: 93.4% -- replaced with /var/log/ceph/bluestore_tool.43481.log.gz 2024-09-16T03:42:48.242 INFO:teuthology.orchestra.run.smithi119.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T03:42:48.277 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T03:42:48.277 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56288.log 2024-09-16T03:42:48.277 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.50472.log 2024-09-16T03:42:48.277 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.56288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48226.log 2024-09-16T03:42:48.277 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56288.log.gz 2024-09-16T03:42:48.277 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/bluestore_tool.50472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40939.log 2024-09-16T03:42:48.277 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.48226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52755.log 2024-09-16T03:42:48.277 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48226.log.gz 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44244.log 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40939.log.gz 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52755.log.gz 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41396.log 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53810.log 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.44244.log.gz 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41396.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.41396.log.gz /var/log/ceph/ceph-client.admin.43107.log 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53810.log.gz 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39810.log 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55700.log 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43107.log.gz 2024-09-16T03:42:48.278 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39810.log.gz 2024-09-16T03:42:48.279 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54506.log 2024-09-16T03:42:48.279 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56162.log 2024-09-16T03:42:48.279 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55700.log.gz 2024-09-16T03:42:48.279 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54506.log.gz 2024-09-16T03:42:48.279 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55720.log 2024-09-16T03:42:48.279 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39822.log 2024-09-16T03:42:48.279 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56162.log.gz 2024-09-16T03:42:48.279 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55720.log.gz 2024-09-16T03:42:48.279 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41393.log 2024-09-16T03:42:48.279 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45899.log 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39822.log.gz 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41393.log.gz 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr: 93.5% -- replaced with /var/log/ceph/bluestore_tool.50472.log.gz 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56372.log 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr:gzip/var/log/ceph/ceph-client.admin.45899.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.54523.log 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45899.log.gz 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40981.log 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56372.log.gz 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54509.log 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54523.log.gz 2024-09-16T03:42:48.280 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40981.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.40981.log.gz /var/log/ceph/ceph-client.admin.39762.log 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42626.log 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54509.log.gz 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39762.log.gz 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33250.log 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42626.log.gz 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39654.log 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.33250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33250.log.gz 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54514.log 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40298.log 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39654.log.gz 2024-09-16T03:42:48.281 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54514.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.54514.log.gz -5 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55130.log 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38881.log 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40298.log.gz 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55136.log 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55130.log.gz 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38881.log.gz 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55868.log 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55136.log.gz 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53222.log 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56624.log 2024-09-16T03:42:48.282 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55868.log.gz 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53222.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.53222.log.gz --verbose 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr: -- /var/log/ceph/ceph-client.admin.45247.log 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39177.log 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56624.log.gz 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54508.log 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.45247.log.gz 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57422.log 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39177.log.gz 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54508.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.54508.log.gz --verbose 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr: -- /var/log/ceph/ceph-client.admin.56792.log 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50884.log 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57422.log.gz 2024-09-16T03:42:48.283 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56750.log 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56792.log.gz 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38797.log 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.50884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50884.log.gz 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57170.log 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56750.log.gz 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42030.log 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38797.log.gz 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43242.log 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57170.log.gz 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44357.log 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42030.log.gz 2024-09-16T03:42:48.284 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55952.log 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.43242.log.gz 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44315.log 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44357.log.gz 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52671.log 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55952.log.gz 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44315.log.gz 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55636.log 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48880.log 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52671.log.gz 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55636.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55636.log.gz 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.48880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40690.log 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48880.log.gz 2024-09-16T03:42:48.285 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54146.log 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39223.log 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40690.log.gz 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55910.log 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54146.log.gz 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39218.log 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39223.log.gz 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54515.log 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55910.log.gz 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39218.log.gz 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42050.log 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44022.log 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54515.log.gz 2024-09-16T03:42:48.286 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42050.log.gz 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41864.log 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40483.log 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44022.log.gz 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41864.log.gz 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.48482.log 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44955.log 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40483.log.gz 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/objectstore_tool.48482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49995.log 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41390.log 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44955.log.gz 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49995.log.gz 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57380.log 2024-09-16T03:42:48.287 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38667.log 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41390.log.gz 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43763.log 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57380.log.gz 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38667.log.gz 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41395.log 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55826.log 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43763.log.gz 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41395.log: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.41395.log.gz 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56876.log 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46451.log 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55826.log.gz 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56876.log.gz 2024-09-16T03:42:48.288 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40678.log 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53096.log 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46451.log.gz 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40678.log.gz 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53516.log 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52970.log 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53096.log.gz 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53516.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.53516.log.gz /var/log/ceph/ceph-client.admin.40689.log 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43222.log 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52970.log.gz 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40689.log.gz 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55134.log 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46369.log 2024-09-16T03:42:48.289 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43222.log.gz 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55134.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.55134.log.gz -5 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr: --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47164.log 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46369.log.gz 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39729.log 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57338.log 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.47164.log.gz 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39729.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.39214.log 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39729.log.gz 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57338.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57338.log.gz 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56708.log 2024-09-16T03:42:48.290 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39214.log.gz 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56414.log 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56708.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56708.log.gz 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr: -- /var/log/ceph/ceph-client.admin.55616.log 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56414.log.gz 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39212.log 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55616.log.gz 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51998.log 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39212.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39212.log.gz 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53642.log 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38839.log 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51998.log.gz 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39215.log 2024-09-16T03:42:48.291 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53642.log.gz 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51674.log 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38839.log: /var/log/ceph/ceph-client.admin.39215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54524.log 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38839.log.gz 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39215.log.gz 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55334.log 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51674.log.gz 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55163.log 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54524.log.gz 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53600.log 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55334.log.gz 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40659.log 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55163.log.gz 2024-09-16T03:42:48.292 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53936.log 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53600.log.gz 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40659.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40659.log.gz 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41107.log 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53936.log.gz 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55994.log 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41107.log.gz 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52609.log 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55994.log.gz 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39726.log 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52609.log.gz 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53348.log 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39726.log.gz 2024-09-16T03:42:48.293 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53348.log.gz 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39863.log 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41798.log 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39863.log.gz 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57296.log 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41798.log: 25.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.41798.log.gz 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54517.log 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57296.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57296.log.gz 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr: 93.7% -- replaced with /var/log/ceph/objectstore_tool.48482.log.gz 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54517.log.gz 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53264.log 2024-09-16T03:42:48.294 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42553.log 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52886.log 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53264.log.gz 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54513.log 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr: 25.6% -- replaced with /var/log/ceph/ceph-client.admin.42553.log.gz 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51323.log 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52886.log.gz 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45431.log 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54513.log.gz 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51323.log.gz 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46540.log 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54020.log 2024-09-16T03:42:48.295 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45431.log.gz 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46540.log.gz 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39192.log 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.49364.log 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54020.log.gz 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54481.log 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39192.log.gz 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/objectstore_tool.49364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52802.log 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41023.log 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54481.log.gz 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41957.log 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52802.log.gz 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41023.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.50861.log 2024-09-16T03:42:48.296 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41023.log.gz 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53012.log 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr: 25.2% -- replaced with /var/log/ceph/ceph-client.admin.41957.log.gz 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/bluestore_tool.50861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51847.log 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53012.log.gz 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54062.log 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51847.log.gz 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39194.log 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54062.log.gz 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55049.log 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39194.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53432.log 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39194.log.gz 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39190.log 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55049.log.gz 2024-09-16T03:42:48.297 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52071.log 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53432.log: /var/log/ceph/ceph-client.admin.39190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53432.log.gz 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39190.log.gz 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52071.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52071.log.gz 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39856.log 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54527.log 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40417.log: /var/log/ceph/ceph-client.admin.39856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57128.log 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39856.log.gz 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47467.log 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54527.log.gz 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5/var/log/ceph/ceph-client.admin.57128.log: --verbose -- /var/log/ceph/ceph-client.admin.40387.log 2024-09-16T03:42:48.298 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57128.log.gz 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47467.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39202.log 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47467.log.gz 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53894.log 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40387.log.gz 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45939.log 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39202.log: /var/log/ceph/ceph-client.admin.53894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39202.log.gz 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42858.log 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53894.log.gz 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54909.log 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.45939.log.gz 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54188.log 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42858.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53054.log 2024-09-16T03:42:48.299 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54909.log: -- replaced with /var/log/ceph/ceph-client.admin.42858.log.gz 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54909.log.gz 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42688.log 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54188.log.gz 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54498.log 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53054.log.gz 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42688.log: /var/log/ceph/ceph-client.admin.52713.log 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42688.log.gz 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54498.log.gz 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39198.log 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43807.log 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52713.log: 93.3% -- replaced with /var/log/ceph/bluestore_tool.50861.log.gz/var/log/ceph/ceph-client.admin.39198.log: gzip 0.0% 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.52713.log.gz -- 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr: /var/log/ceph/ceph-client.admin.57504.log 2024-09-16T03:42:48.300 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39198.log.gz 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56960.log 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43807.log.gz 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27344.log 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57504.log.gz 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46411.log 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56960.log.gz 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph.tmp-client.admin.27344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42195.log 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27344.log.gz 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46411.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46411.log.gz 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53978.log 2024-09-16T03:42:48.301 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42195.log.gz 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39188.log 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53978.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.45552.log 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53978.log.gz 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39188.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39188.log.gz 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39632.log 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45552.log.gz 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55312.log 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39632.log.gz 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56330.log 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55312.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.57212.log 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55312.log.gz 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51931.log 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56330.log.gz 2024-09-16T03:42:48.302 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5/var/log/ceph/ceph-client.admin.57212.log: --verbose -- /var/log/ceph/objectstore_tool.47420.log 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57212.log.gz 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51931.log: 25.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52844.log 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51931.log.gz 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/objectstore_tool.47420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56666.log 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56204.log 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52844.log.gz 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56666.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43148.log 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56666.log.gz 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52129.log 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56204.log: /var/log/ceph/ceph-client.admin.43148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56204.log.gz 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43148.log.gz 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55574.log 2024-09-16T03:42:48.303 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53474.log 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52129.log.gz 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55574.log.gz 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42945.log 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53390.log 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53474.log.gz 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42945.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42511.log 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42945.log.gz 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55143.log 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53390.log.gz 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42511.log: 93.4%gzip -- replaced with /var/log/ceph/objectstore_tool.49364.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.39873.log 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42511.log.gz 2024-09-16T03:42:48.304 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56582.log 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55143.log.gz 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5/var/log/ceph/ceph-client.admin.39873.log: --verbose -- /var/log/ceph/ceph-client.admin.39213.log 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39873.log.gz 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56582.log.gz 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53684.log 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39213.log.gz 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54482.log 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53684.log.gz 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54482.log: 28.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53306.log 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54482.log.gz 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55784.log 2024-09-16T03:42:48.305 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53306.log.gz 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39792.log 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.55784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55784.log.gz 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51583.log 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39792.log.gz 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44448.log 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr: 85.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51583.log.gz 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40347.log 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44448.log.gz 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55011.log 2024-09-16T03:42:48.306 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40691.log 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55011.log: /var/log/ceph/ceph-client.admin.40347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40347.log.gz 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55011.log.gz 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38694.log 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49107.log 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40691.log.gz 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38694.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40140.log 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40677.log 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49107.log: /var/log/ceph/ceph-client.admin.40140.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49107.log.gz 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40140.log.gz 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53558.log 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40677.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57254.log 2024-09-16T03:42:48.307 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40677.log.gz 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53558.log.gz 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46155.log 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.50391.log 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57254.log.gz 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46155.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57002.log 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46155.log.gz 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/bluestore_tool.50391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43417.log 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57002.log: 85.3% -- replaced with /var/log/ceph/bluestore_tool.50391.log.gz 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57002.log.gz 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42397.log 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40383.log 2024-09-16T03:42:48.308 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43417.log: 24.3% -- replaced with /var/log/ceph/ceph-client.admin.43417.log.gz 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42397.log: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.42397.log.gz 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57464.log 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39816.log 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40383.log: /var/log/ceph/ceph-client.admin.57464.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40383.log.gz 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57464.log.gz 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45360.log 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39816.log: -- /var/log/ceph/ceph-client.admin.40032.log 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39816.log.gz 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45360.log.gz 93.9% -- replaced with /var/log/ceph/objectstore_tool.47420.log.gz 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40032.log.gz 2024-09-16T03:42:48.309 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45825.log 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45318.log 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56834.log 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45825.log: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.45825.log.gz 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45318.log.gz 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56036.log 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56834.log.gz 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54104.log 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40267.log 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56036.log: /var/log/ceph/ceph-client.admin.54104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54104.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56036.log.gz 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55065.log 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44827.log 2024-09-16T03:42:48.310 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40267.log.gz 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55065.log.gzgzip 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52567.log 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44894.log 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52567.log: /var/log/ceph/ceph-client.admin.44827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44827.log.gz 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52567.log.gz 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41253.log 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44894.log: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.44894.log.gz 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44569.log 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44781.log 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41253.log.gz 2024-09-16T03:42:48.311 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56246.log 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr: 25.2% -- replaced with /var/log/ceph/ceph-client.admin.44569.log.gz 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56456.log 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.44781.log.gz 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56120.log 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56246.log.gz 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56456.log.gz 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33516.log 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33722.log 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56120.log.gz 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.33516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33516.log.gz 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54530.log 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.46558.log 2024-09-16T03:42:48.312 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54530.log: /var/log/ceph/ceph-client.admin.33722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33722.log.gz 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54530.log.gz 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54975.log 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/objectstore_tool.46558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53180.log 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54975.log.gz 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38611.log 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53180.log.gz 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45784.log 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38611.log.gz 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49771.log 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45784.log.gz 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56498.log 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49771.log.gz 2024-09-16T03:42:48.313 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52928.log 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56498.log.gz 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55055.log 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52928.log.gz 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.51362.log 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55055.log.gz 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43889.log 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/bluestore_tool.51362.log: 92.5% -- replaced with /var/log/ceph/bluestore_tool.51362.log.gz 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55135.log 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43889.log.gz 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53138.log 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55135.log.gz 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42113.log 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53138.log.gz 2024-09-16T03:42:48.314 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54486.log 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42113.log.gz 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56078.log 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54486.log.gz 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39191.log 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56078.log.gz 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43849.log 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39191.log.gz 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57086.log 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43849.log.gz 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43498.log 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57086.log.gz 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55118.log 2024-09-16T03:42:48.315 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43498.log: 26.0% -- replaced with /var/log/ceph/ceph-client.admin.43498.log.gz 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56540.log 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55118.log.gz 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42728.log 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56540.log.gz 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53726.log 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42728.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.42728.log.gz 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40682.log 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53726.log.gz 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56918.log 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40682.log.gz 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40687.log 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56918.log.gz 2024-09-16T03:42:48.316 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55120.log 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40687.log.gz 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53852.log 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55120.log.gz 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54512.log 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53852.log.gz 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39217.log 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54512.log.gz 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39674.log 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39217.log.gz 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41191.log 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39674.log.gz 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53768.log 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41191.log.gz 2024-09-16T03:42:48.317 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46114.log 2024-09-16T03:42:48.318 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53768.log.gz 2024-09-16T03:42:48.318 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57044.log 2024-09-16T03:42:48.318 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46114.log: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.46114.log.gz 2024-09-16T03:42:48.318 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41871.log 2024-09-16T03:42:48.318 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57044.log.gz 2024-09-16T03:42:48.318 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43692.log 2024-09-16T03:42:48.318 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41871.log.gz 2024-09-16T03:42:48.318 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43692.log: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.43692.log.gz 2024-09-16T03:42:48.318 INFO:teuthology.orchestra.run.smithi076.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T03:42:48.319 INFO:teuthology.orchestra.run.smithi076.stderr: 94.2% -- replaced with /var/log/ceph/objectstore_tool.46558.log.gz 2024-09-16T03:42:48.319 INFO:teuthology.orchestra.run.smithi103.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T03:42:48.503 INFO:teuthology.orchestra.run.smithi119.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T03:42:48.513 INFO:teuthology.orchestra.run.smithi121.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T03:42:48.525 INFO:teuthology.orchestra.run.smithi076.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T03:42:48.584 INFO:teuthology.orchestra.run.smithi119.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-16T03:42:48.648 INFO:teuthology.orchestra.run.smithi119.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-16T03:42:48.651 INFO:teuthology.orchestra.run.smithi076.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-16T03:42:48.802 INFO:teuthology.orchestra.run.smithi103.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T03:42:48.804 INFO:teuthology.orchestra.run.smithi103.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-16T03:42:48.900 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T03:42:48.926 INFO:teuthology.orchestra.run.smithi121.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-16T03:42:48.940 INFO:teuthology.orchestra.run.smithi121.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T03:42:48.949 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-16T03:42:49.041 INFO:teuthology.orchestra.run.smithi076.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T03:42:49.084 INFO:teuthology.orchestra.run.smithi103.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-16T03:42:49.086 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T03:42:49.086 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m0.977s 2024-09-16T03:42:49.086 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m3.244s 2024-09-16T03:42:49.086 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.136s 2024-09-16T03:42:49.126 INFO:teuthology.orchestra.run.smithi119.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T03:42:49.128 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-09-16T03:42:49.128 INFO:teuthology.orchestra.run.smithi119.stderr:real 0m1.016s 2024-09-16T03:42:49.128 INFO:teuthology.orchestra.run.smithi119.stderr:user 0m2.418s 2024-09-16T03:42:49.128 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m0.093s 2024-09-16T03:42:49.184 INFO:teuthology.orchestra.run.smithi121.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-16T03:42:49.185 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-16T03:42:49.185 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m1.070s 2024-09-16T03:42:49.185 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m2.959s 2024-09-16T03:42:49.185 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m0.126s 2024-09-16T03:42:49.265 INFO:teuthology.orchestra.run.smithi076.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T03:42:49.267 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T03:42:49.267 INFO:teuthology.orchestra.run.smithi076.stderr:real 0m1.158s 2024-09-16T03:42:49.267 INFO:teuthology.orchestra.run.smithi076.stderr:user 0m3.798s 2024-09-16T03:42:49.267 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m0.310s 2024-09-16T03:42:49.267 INFO:tasks.ceph:Archiving logs... 2024-09-16T03:42:49.268 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi076/log 2024-09-16T03:42:49.269 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T03:42:50.097 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi103/log 2024-09-16T03:42:50.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T03:42:50.634 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi119/log 2024-09-16T03:42:50.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T03:42:51.087 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi121/log 2024-09-16T03:42:51.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T03:42:51.628 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T03:42:51.637 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T03:42:51.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T03:42:51.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T03:42:51.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T03:42:51.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T03:42:51.773 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T03:42:51.781 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T03:42:51.782 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:42:51.784 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:42:51.787 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:42:51.789 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:42:51.799 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:42:51.801 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:42:51.804 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:42:51.805 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:42:51.805 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-09-16T03:42:51.805 INFO:teuthology.orchestra.run.smithi076.stdout:^* hv01.front.sepia.ceph.com 3 6 377 24 -604us[ -580us] +/- 48ms 2024-09-16T03:42:51.805 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 22 +949us[ +949us] +/- 67ms 2024-09-16T03:42:51.805 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -77us[ -77us] +/- 89ms 2024-09-16T03:42:51.805 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:42:51.807 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:42:51.807 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:42:51.808 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-09-16T03:42:51.808 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv01.front.sepia.ceph.com 3 6 377 22 -593us[ -568us] +/- 48ms 2024-09-16T03:42:51.808 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 24 +985us[+1010us] +/- 67ms 2024-09-16T03:42:51.808 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -89us[ -89us] +/- 89ms 2024-09-16T03:42:51.808 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:42:51.809 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:42:51.809 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-09-16T03:42:51.810 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv01.front.sepia.ceph.com 3 6 377 22 -593us[ -579us] +/- 48ms 2024-09-16T03:42:51.810 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 22 +972us[ +972us] +/- 67ms 2024-09-16T03:42:51.810 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -67us[ -53us] +/- 89ms 2024-09-16T03:42:51.810 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:42:51.812 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:42:51.812 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-09-16T03:42:51.812 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv01.front.sepia.ceph.com 3 6 377 22 -593us[ -596us] +/- 48ms 2024-09-16T03:42:51.812 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 22 +969us[ +969us] +/- 67ms 2024-09-16T03:42:51.812 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 -70us[ -73us] +/- 89ms 2024-09-16T03:42:51.812 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:42:51.813 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T03:42:51.823 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T03:42:51.823 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T03:42:51.831 DEBUG:teuthology.orchestra.run.smithi076:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T03:42:51.923 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T03:42:51.996 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T03:42:52.067 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T03:42:52.138 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:42:52.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:52.172 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:42:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:52.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:42:52.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:52.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:42:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:52.266 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T03:42:52.276 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T03:42:52.286 INFO:teuthology.task.internal:Duration was 651.919321 seconds 2024-09-16T03:42:52.286 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T03:42:52.295 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T03:42:52.295 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T03:42:52.298 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T03:42:52.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T03:42:52.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T03:42:52.340 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:42:52.342 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:42:52.342 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:42:52.348 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:42:52.801 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T03:42:52.801 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T03:42:52.801 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T03:42:52.826 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T03:42:52.826 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T03:42:52.861 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-09-16T03:42:52.861 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T03:42:52.883 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-09-16T03:42:52.883 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T03:42:52.908 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T03:42:52.909 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T03:42:52.912 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T03:42:52.914 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T03:42:52.925 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T03:42:52.979 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T03:42:52.979 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T03:42:52.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T03:42:52.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T03:42:52.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T03:42:53.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T03:42:53.138 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T03:42:53.138 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T03:42:53.164 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T03:42:53.190 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T03:42:53.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T03:42:53.240 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T03:42:53.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:53.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:53.255 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:53.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:53.273 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2024-09-16T03:42:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-09-16T03:42:53.279 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-09-16T03:42:53.303 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-09-16T03:42:53.345 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:53.373 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:53.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:53.388 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:53.403 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:53.419 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T03:42:53.435 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T03:42:53.437 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi076 2024-09-16T03:42:53.438 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T03:42:53.477 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi103 2024-09-16T03:42:53.478 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T03:42:53.518 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi119 2024-09-16T03:42:53.519 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T03:42:53.556 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905894/remote/smithi121 2024-09-16T03:42:53.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T03:42:53.593 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T03:42:53.593 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T03:42:53.596 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T03:42:53.598 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T03:42:53.600 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T03:42:53.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T03:42:53.652 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T03:42:53.652 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T03:42:53.664 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T03:42:53.664 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T03:42:53.667 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T03:42:53.669 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T03:42:53.671 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T03:42:53.679 INFO:teuthology.orchestra.run.smithi076.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 03:42 /home/ubuntu/cephtest 2024-09-16T03:42:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 03:42 /home/ubuntu/cephtest 2024-09-16T03:42:53.684 INFO:teuthology.orchestra.run.smithi119.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 03:42 /home/ubuntu/cephtest 2024-09-16T03:42:53.696 INFO:teuthology.orchestra.run.smithi121.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 03:42 /home/ubuntu/cephtest 2024-09-16T03:42:53.697 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T03:42:53.710 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T03:42:53.733 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 651.9193210601807 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T03:42:53.733 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T03:42:53.829 INFO:teuthology.run:pass